TechnicalExpert6 min read
Part II. Architecture and reference configurations
Sending AI signals through the same circuit as the rest of the system: the unified collection plane, a reference Collector configuration and MetricsQL recording rules for latency, errors and cost.
For: engineers and SREs · architectsPrerequisites: Have read part I; know the OpenTelemetry Collector and the PromQL or MetricsQL language.
5. The unified collection plane
Section titled “5. The unified collection plane”An illustrative scenario: the team building the new assistant installs its own tool to follow model calls, next to the monitoring that operations already use for servers and databases. One evening, answers slow down. The AI team sees slow calls; operations see a saturated vector store. It takes them an hour on the phone to realize they are looking at the same incident from two sides.
I recommend not building a GenAI silo. Traces, metrics and logs from LLMs, agents, RAG and MCP should travel through the same circuit as those of existing infrastructure and applications, the way a new department’s mail goes through the same sorting office as the rest of the company. That shared circuit is called the collection plane.
The method depends on no particular tool. VictoriaMetrics, Grafana and Langfuse are one example among others, chosen here because it can be self-hosted and relies on open protocols (OTLP, the OpenTelemetry protocol; remote write, the one used to push metrics into a database). The same roles can be filled with Prometheus, Mimir or Thanos for metrics, Tempo or Jaeger for traces, Loki for logs, Phoenix (under the Elastic License 2.0, a source-available license) or a commercial service such as LangSmith for evaluation. Langfuse has belonged to ClickHouse since January 2026; its core remains under the MIT license.
The diagram follows the path of a signal. The application is instrumented, meaning it carries libraries that describe what it does (OpenLLMetry, OpenInference or the native SDK). These descriptions reach the OpenTelemetry Collector, a middle program that sorts, cleans and forwards. The Collector masks sensitive data, then sends each signal to the store that suits it. Grafana displays the lot.
flowchart TB APP["Application: LLM / Agent / RAG / MCP clients"] INST["OTel instrumentation<br/>OpenLLMetry, OpenInference, native SDK"] COL["OpenTelemetry Collector<br/>masking, attributes, spanmetrics"] MET["Metrics<br/>VictoriaMetrics<br/>(DCGM, vLLM)"] TRA["Traces<br/>OTLP backend"] LOG["Logs<br/>VictoriaLogs / Loki"] CON["Content opt-in<br/>encrypted object store"] EVAL["Evaluation layer<br/>Langfuse / Phoenix"] GRAF["Grafana<br/>dashboards, alerting, SLOs"] APP --> INST --> COL COL --> MET COL --> TRA COL --> LOG COL --> CON MET --> GRAF TRA --> GRAF LOG --> GRAF TRA --> EVAL CON --> EVAL EVAL --> GRAF
6. Reference collector configuration
Section titled “6. Reference collector configuration”The Collector works like a checkpoint at the entrance of a warehouse. It receives the parcels (the signals), removes what must not come in (email addresses, card numbers), then routes each parcel to the right aisle.
The proposed configuration, to adapt, organizes this work into three processing chains called pipelines: one for traces, one for metrics, one for logs. It masks sensitive data before anything leaves. It also builds metrics out of traces, through a component named spanmetrics: each trace step (a span) is counted and timed, then filed by model, operation type, agent and error type. It deliberately sticks to these low-cardinality dimensions, which take few distinct values, so the number of stored series does not explode.
One detail matters. If an instrumentation captures message content by mistake, a filter strips it from spans before export. Opt-in content goes through encrypted external storage, never through the trace backend.
For technical teams: the annotated otel-collector.yaml file
# otel-collector.yaml (reference excerpt)receivers: otlp: protocols: grpc: { endpoint: 0.0.0.0:4317 } http: { endpoint: 0.0.0.0:4318 }
processors: batch: timeout: 5s send_batch_size: 1000
# Masking: strip or hash sensitive content before any export. redaction: allow_all_keys: true blocked_values: - "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}" # emails - "\\b(?:\\d[ -]*?){13,16}\\b" # card numbers summary: info
# Safety net on traces: if an instrumentation captures content by # mistake, strip it from spans before export. Opt-in content goes to the # encrypted external store, never to the trace backend. # Metrics do not need this filter: spanmetrics only keeps the dimensions # listed below. attributes/clean: actions: - key: gen_ai.input.messages action: delete - key: gen_ai.output.messages action: delete
connectors: # Derive metrics from spans, low-cardinality dimensions. spanmetrics: dimensions: - name: gen_ai.request.model - name: gen_ai.operation.name - name: gen_ai.agent.name - name: error.type
exporters: # Metrics to VictoriaMetrics (remote write endpoint). prometheusremotewrite: endpoint: http://victoriametrics:8428/api/v1/write # Traces to the OTLP backend. otlp/traces: endpoint: tempo:4317 tls: { insecure: true } # Logs to VictoriaLogs or Loki. otlphttp/logs: logs_endpoint: http://victorialogs:9428/insert/opentelemetry/v1/logs
service: pipelines: traces: receivers: [otlp] processors: [attributes/clean, redaction, batch] exporters: [otlp/traces, spanmetrics] metrics: receivers: [otlp, spanmetrics] processors: [batch] exporters: [prometheusremotewrite] logs: receivers: [otlp] processors: [redaction, batch] exporters: [otlphttp/logs]7. MetricsQL recording rules
Section titled “7. MetricsQL recording rules”Computing the latency of every call from the past month each time a chart loads is like recounting the whole till every time you want today’s takings. A recording rule does that computation at a regular interval and stores the result under a new name. Dashboards and alerts then read the ready-made result.
The method proposes five rules, to set in VictoriaMetrics or a compatible Prometheus, written in MetricsQL (VictoriaMetrics’ query language, close to PromQL). They compute p95 latency per model, meaning the time under which 95% of calls complete. They also compute TTFT at the same percentile for streaming calls, the error ratio per error type, the estimated cost per second and the average number of input tokens per request. That last figure catches prompt bloat: if it doubles in a month, someone has added a lot of text somewhere.
The source metrics are those of the OTel GenAI conventions 1.40 and 1.41 (gen_ai.client.operation.duration, gen_ai.client.token.usage, plus gen_ai.client.operation.time_to_first_chunk from 1.41 onwards). Their names change on the way. By default (translation_strategy: UnderscoreEscapingWithSuffixes), the prometheusremotewrite exporter replaces dots with underscores and appends the unit suffix. So gen_ai.client.operation.duration (unit s) becomes gen_ai_client_operation_duration_seconds_bucket, _sum and _count. The {token} unit is only an annotation and adds no suffix: gen_ai_client_token_usage_sum stays as is. If you choose UnderscoreEscapingWithoutSuffixes, remove _seconds from the rules. Take prices from your own dated table (annex D).
For technical teams: the rules.yaml file
# rules.yaml (reference excerpt, MetricsQL syntax)groups: - name: genai interval: 30s rules: # p95 latency per model: operation duration. - record: genai:operation_duration:p95 expr: | histogram_quantile(0.95, sum(rate(gen_ai_client_operation_duration_seconds_bucket[5m])) by (le, gen_ai_request_model))
# p95 TTFT per model: time to first chunk, streaming calls only. - record: genai:time_to_first_chunk:p95 expr: | histogram_quantile(0.95, sum(rate(gen_ai_client_operation_time_to_first_chunk_seconds_bucket[5m])) by (le, gen_ai_request_model))
# Error ratio per error type. - record: genai:error_ratio expr: | sum(rate(gen_ai_client_operation_duration_seconds_count{error_type!=""}[5m])) by (error_type) / ignoring(error_type) group_left sum(rate(gen_ai_client_operation_duration_seconds_count[5m]))
# Estimated cost rate (euros per second), via tokens and prices injected as labels. # Divide by the request rate to get a cost per request. # price_in and price_out come from a price recording rule or a relabel. - record: genai:cost_rate:eur_per_second expr: | (sum(rate(gen_ai_client_token_usage_sum{gen_ai_token_type="input"}[5m])) by (gen_ai_request_model) * on(gen_ai_request_model) group_left price_in_eur_per_token) + (sum(rate(gen_ai_client_token_usage_sum{gen_ai_token_type="output"}[5m])) by (gen_ai_request_model) * on(gen_ai_request_model) group_left price_out_eur_per_token)
# Average input tokens per request (prompt bloat detection). - record: genai:input_tokens_per_request:avg expr: | sum(rate(gen_ai_client_token_usage_sum{gen_ai_token_type="input"}[1h])) by (gen_ai_request_model) / sum(rate(gen_ai_client_operation_duration_seconds_count[1h])) by (gen_ai_request_model)Where does the price come from?
Section titled “Where does the price come from?”The cost rule multiplies tokens by a price. That price does not appear by magic: you keep it in a small per-model series, like a price tag on each shelf, kept in sync with your own dated price table (annex D). Prices change fast and vary by contract. This guide therefore gives none; use your provider’s dated price list and replace <p_in> and <p_out> with your prices in euros per token.
# <p_in> and <p_out>: fill in from your provider's dated price list.- record: price_in_eur_per_token labels: { gen_ai_request_model: "demo-llm" } expr: vector(<p_in>)- record: price_out_eur_per_token labels: { gen_ai_request_model: "demo-llm" } expr: vector(<p_out>)The cost rule’s group_left join reads these series. As long as no price series exists for a model, the join returns nothing and the rule emits no cost for that model; the token rules (genai:input_tokens_per_request:avg) remain available. In production, a job reads a versioned price table to produce the two series. The public genaiotel lab follows a variant: it computes the cost of each span in the Collector (with OTTL, its transformation language) from a price table, then a vmalert rule derives the cost series.
Dashboards
Section titled “Dashboards”I recommend one Grafana dashboard per layer, linked to each other by trace_id. The main panels show p95 latency, error ratio by error.type, the distribution of finish_reasons, cost per request and per feature, plus input tokens per request over time.
The design rule is simple: every panel must inform a decision, otherwise it goes. A panel that represents an SLO has its alert (see the guide, 9.5).
For Monday morning
Section titled “For Monday morning”Check that calls to your models go through the same Collector as the rest of your services. If they do not, that is the first job, before any new tool. Then set up the input-tokens-per-request rule: it needs no price and reveals a swelling prompt very early.
Revised on 2 October 2026: prices removed, metric names fixed (_seconds suffix added by the exporter), TTFT rule added, content filter moved to the traces pipeline, pointer to the public genaiotel lab, mention of the acquisition of Langfuse by ClickHouse. Token metrics are evolving in the semantic-conventions-genai repository (see Part I).
Revised on 4 October 2026: the dashboard design rule points to guide 9.5, the corresponding pitfall having left Part VIII.
Revised on 25 October 2026: page rewritten for a wider audience (opening scenario, terms explained, configurations folded away), with no change of substance.