Skip to content

TechnicalPractitioner7 min read

Labs, go-live checklist and MetricsQL cheat sheet

Four practical labs on VictoriaMetrics for LLM workloads, a production go-live checklist, a MetricsQL cheat sheet for LLMs and the official references.

For: engineers and SREsPrerequisites: Have read lessons 1 to 5 of the course and have Docker with Compose v2.

Reading a configuration is no substitute for the moment an alert actually fires on your own machine. The four labs below follow the lessons in order: deploy, query, alert, speed up. They run on a laptop, with a simulator that produces believable LLM traffic without calling any model.

Lab 1: deploy the stack and ingest first metrics

Section titled “Lab 1: deploy the stack and ingest first metrics”

This first lab deploys the complete Docker Compose stack (VictoriaMetrics, OTel Collector, vmagent, vmalert, Alertmanager, Grafana, simulator), then checks that metrics reach vmui and Grafana. Allow 30 minutes, beginner level. You need Docker with Compose v2 and about 8 GB of free RAM (the stack runs nine containers).

  1. Clone the repository and start the stack:

    Terminal window
    git clone https://forge.erythix.tech/labstraining/labs/VMLLM.git
    cd VMLLM
    docker compose up -d --build
    docker compose ps # check that all services are running
  2. The simulator starts with the stack (llm-simulator service, 10 requests per second, 3 models) and exposes its metrics on port 9100. Check that it produces series:

    Terminal window
    curl -s http://localhost:9100/metrics | grep llm_requests_total | head
  3. Check ingestion in vmui (http://localhost:8428/vmui) with the query:

    sum by (model) (rate(llm_requests_total[1m])) * 60
  4. Open Grafana (http://localhost:3000). Dashboards are provisioned at startup: open the production dashboard (uid llm-prod), defined in grafana/dashboards/dashboard-llm-prod.json.

Using the simulator’s metrics, you write the five core production queries, validate them in vmui, then add them to a Grafana dashboard. Allow 40 minutes, intermediate level. The solution is in solutions/lab2-queries.md in the repository.

  1. Total cost by model over the last 24 hours.
  2. TTFT P95 by provider (histogram_quantile).
  3. RAG retrieval score anomaly (z-score against a 7-day baseline, see module 7; the baseline needs seven days of history, so on a fresh stack the z-score is partial).
  4. Top 5 tenants by cost (topk + increase).
  5. GPU efficiency: output tokens per joule (division between metrics).

This lab replays the scenario from lesson 5: an index whose quality drops silently. You configure the RAGIndexDrift alert in vmalert, test it by injecting a score degradation, then check the notification in a test Slack channel. Allow 25 minutes, plus the alert’s 15-minute for:, intermediate level.

  1. Adapt the repository’s alerts/llm-quality.yml and alertmanager.yml with your test Slack webhook URL.
  2. Reload the rules with a GET request: curl http://localhost:8880/-/reload (or docker compose restart vmalert). Reload Alertmanager with docker compose restart alertmanager.
  3. Inject a score degradation: make lab3-drift (or bash scripts/triggers/01_rag_index_drift.sh). The script stops the nominal simulator and starts one with --score-drift 0.5, which halves the scores.
  4. In the vmalert UI (http://localhost:8880), check that the alert goes to pending then fires after 15 minutes.
  5. Confirm receipt in Slack, then restore the nominal simulator: make lab3-restore (or bash scripts/triggers/_restore.sh).

You add the main recording rules and compare, through the VictoriaMetrics API, query response times with and without them. Allow 20 minutes, advanced level. The solution is in solutions/lab4-recording-rules.md in the repository (ignore its speedup figures, see the note on the VMLLM lab).

  1. Copy the rules of module 10 into alerts/recording-rules.yml (the repository contains a similar version), then reload vmalert: curl http://localhost:8880/-/reload.

  2. Wait two or three evaluation intervals, then check in vmui that the llm:request_duration_ms:p99_5m series exists.

  3. Time the full query:

    Terminal window
    time curl -sG 'http://localhost:8428/api/v1/query' \
    --data-urlencode 'query=histogram_quantile(0.99, sum by (le, model, pipeline_id) (rate(llm_request_duration_ms_bucket[5m])))' \
    > /dev/null
  4. Time the pre-computed series:

    Terminal window
    time curl -sG 'http://localhost:8428/api/v1/query' \
    --data-urlencode 'query=llm:request_duration_ms:p99_5m' \
    > /dev/null
  5. Repeat each measurement about ten times and compare the medians. Do the same with increase(llm_cost_usd_total[24h]) and llm:cost_usd:increase24h.

  6. Interpret: the simulator’s cardinality is low (a few hundred series), so the absolute gap will be small. It grows with the number of series the full query reads; this is the ratio to measure on your own instance before rolling the rules out widely.

A checklist is meant to be ticked, not skimmed. This one gathers the points from lessons 1 to 6 that, once forgotten, tend to surface in the middle of an incident. The runbook mentioned below is the procedure sheet the on-call engineer opens when the alert goes off. D+7 and D+30 are the checks to run one week and one month after go-live.

  • VictoriaMetrics is deployed with -storageDataPath on a persistent SSD.
  • Retention is set: -retentionPeriod on each instance and -retentionFilter only if you use the Enterprise version (otherwise, separate instances per retention).
  • vmbackup is configured to S3-compatible storage (restore test passed).
  • vmagent or the OTel Collector ingests the four metric categories (quality, latency, cost, GPU).
  • The six production vmalert rules are loaded and tested (test alert received in Slack or Teams).
  • At least two recording rules are active for costs.
  • The production Grafana dashboard is imported and template variables work.
  • vmauth is configured with authentication (basic auth or bearer token; accepting mTLS connections is a vmauth Enterprise feature), never unauthenticated access.
  • Total cardinality is measured and documented (/api/v1/status/tsdb).
  • Risky labels (trace_id, session_id) are dropped by relabelling.
  • A VictoriaMetricsDown self-monitoring alert is active.
  • An incident runbook exists (who to call, how to restore, how to scale).
  • Read-only Grafana users exist for non-technical roles (management, finance).
  • Cardinality has stabilized (no linear growth).
  • Disk consumption is measured and projected to 90 days.
  • At least one real alert has been received and routing confirmed.
  • P99 latency of Grafana queries is measured and compared with the target you set (for example under 1 s).
  • A vmbackup restore test has been run on a test instance.
  • Alert thresholds have been reviewed with the teams (false positives?).
  • Custom MetricsQL queries used by the teams are documented.

Appendix B: MetricsQL cheat sheet for LLMs

Section titled “Appendix B: MetricsQL cheat sheet for LLMs”

The queries from lessons 4 and 5, gathered in one place for copying. Each one assumes the metric names from lesson 3.

# Hourly cost per model
sum by (model) (increase(llm_cost_usd_total[1h]))
# Projected daily cost
sum(rate(llm_cost_usd_total[30m])) * 86400
# Top 10 tenants over 24h
topk(10, sum by (tenant_id) (increase(llm_cost_usd_total[24h])))
# Average cost per request (recording rule recommended)
sum(rate(llm_cost_usd_total[5m])) / sum(rate(llm_requests_total[5m]))
# P99 end-to-end latency per model
histogram_quantile(0.99, sum by (le, model) (rate(llm_request_duration_ms_bucket[5m])))
# TTFT P95 (streaming)
histogram_quantile(0.95, sum by (le, model) (rate(llm_ttft_ms_bucket[5m])))
# Global error rate
sum(rate(llm_errors_total[5m])) / sum(rate(llm_requests_total[5m]))
# Output token throughput per model (tokens/s)
sum by (model) (rate(llm_tokens_total{type='output'}[1m]))
# Retrieval score z-score against a 7-day baseline
# (replaces anomaly_score(), which is not a MetricsQL function)
(
avg by (index_id) (rag_retrieval_score_avg)
- avg_over_time(avg by (index_id) (rag_retrieval_score_avg)[7d:1h])
)
/ stddev_over_time(avg by (index_id) (rag_retrieval_score_avg)[7d:1h])
# If you run vmanomaly (Enterprise): query the metric it writes
anomaly_score > 1
# P95 context pressure per pipeline
histogram_quantile(0.95, sum by (le, pipeline_id) (rate(rag_context_pressure_ratio_bucket[5m])))
# Share of dropped chunks (requires a rag_chunks_retrieved_total counter)
sum by (pipeline_id) (rate(rag_chunks_dropped_total[5m]))
/ sum by (pipeline_id) (rate(rag_chunks_retrieved_total[5m]))
# Average GPU utilization per node and GPU
avg by (node, gpu) (DCGM_FI_DEV_GPU_UTIL)
# VRAM saturation (alert before OOM)
DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)
# Output tokens per joule (global; DCGM series have no LLM model label)
sum(rate(llm_tokens_total{type='output'}[5m])) / sum(DCGM_FI_DEV_POWER_USAGE)
# vLLM queue
vllm:num_requests_waiting
# Robust median (ignores spikes)
median_over_time(rag_retrieval_score_avg[30m])
# Open, high, low, close of RAG scores (candlestick visualization)
rollup_candlestick(rag_retrieval_score_avg[1h])
# Near-zero traffic: the slope of a counter is a per-second rate
abs(deriv(llm_requests_total[10m])) < 0.01
# Week-over-week comparison
sum(rate(llm_cost_usd_total[1h]))
/ sum(rate(llm_cost_usd_total[1h] offset 7d))
ResourceURL
VictoriaMetrics documentationdocs.victoriametrics.com
MetricsQL referencedocs.victoriametrics.com/metricsql
vmagentdocs.victoriametrics.com/vmagent
vmalertdocs.victoriametrics.com/vmalert
vmauthdocs.victoriametrics.com/vmauth
vmanomalydocs.victoriametrics.com/anomaly-detection
VictoriaMetrics source codegithub.com/VictoriaMetrics/VictoriaMetrics
ToolURL
OTel Collector Contribgithub.com/open-telemetry/opentelemetry-collector-contrib
DCGM exporter (GPU)github.com/NVIDIA/dcgm-exporter
vLLM metricsdocs.vllm.ai (section “Metrics”)
LiteLLM proxy metricsdocs.litellm.ai/docs/proxy/prometheus
Grafana VictoriaMetrics plugingrafana.com/grafana/plugins/victoriametrics-metrics-datasource
prometheus_client for Pythongithub.com/prometheus/client_python