TechnicalPractitioner9 min read
7. Operating the platform
Keeping an AI observability platform running: sampling strategy, cost modeling, cardinality management, personal data and data sovereignty, security and organizational ownership.
For: engineers and SREs · architects · team managersPrerequisites: Have read chapters 4 and 5 of the guide.
Illustrative scenario. Three months after setting up the platform, the insurer gets two nasty surprises in the same week. The trace storage bill has doubled. And an internal audit finds that a member pasted their IBAN into a question: it is now readable by the whole team in the trace tool.
Implementation was only the start. Keeping an AI observability platform running takes deliberate choices: what data to keep, what it costs, how to protect it and who looks after it.
7.1 Sampling strategy
Section titled “7.1 Sampling strategy”Tracing every LLM request in full is expensive in storage and processing. Above a few hundred requests per second you have to sample, meaning keep only part of the traces.
There are two ways to decide which ones. Head sampling decides at the start of the request, at random, like a ticket inspector who picks passengers by lot as they board. Tail sampling waits until the request ends and decides by rule, like an inspector who only keeps the disputed tickets once the journey is over. The second costs more memory but guarantees you keep the interesting traces.

The figure simplifies: only scores from synchronous evaluations are available when the sampling decision is made (see below).
Figure 10. Head sampling decides at trace start, randomly. Tail sampling decides at trace end, by policy. Tail sampling preserves errors, slow traces and traces flagged by a synchronous evaluation deterministically.
Recommended policy
Section titled “Recommended policy”I recommend always keeping error traces, abnormally slow traces and traces where a synchronous evaluation is below threshold. A synchronous evaluation runs during the request, such as a toxicity filter or a refusal classifier. For the rest, keep 10 percent of traces at random, to retain a baseline view of normal traffic.
There is one limit to know. The processor can only decide on what the trace contains when its wait time expires: errors, latencies, attributes set during the request. Asynchronous evaluator scores and user feedback arrive much later, so they cannot drive this decision. To keep traces scored after the fact, keep all raw traces on a short retention (a few days). Then promote to long-term storage those that an evaluator scored poorly or that received negative feedback.
For technical teams: the tail_sampling processor and its sizing
Implementation uses the OpenTelemetry Collector tail_sampling processor. Its wait time is decision_wait, 30 seconds by default.
The processor’s memory cost depends on the number of traces held pending (num_traces) and on the decision_wait duration, multiplied by the average trace size: size these two parameters from your throughput.
7.2 Cost modeling
Section titled “7.2 Cost modeling”This section is the site’s reference on the cost of observability itself. The cost of non-observability (the economic exposure of a poorly observed system, with its calculator) is covered in the GenAI method, Part V.
At scale, the cost of AI observability can rival the cost of the LLM calls themselves. It breaks down into three items: storage, processing (the Collector and ingestion) and the judge, which consumes tokens at every evaluation.
Take a worked example. At 100 requests per second, with a 4 KB average payload and 30 days of hot retention, you keep about 988.8 GB, close to 1 TB. That volume is multiplied by a price per GB-month: it is a monthly cost. On the judge side, with 10 percent of requests evaluated and 1,500 tokens per judgment, you get 864,000 judgments per day, or 1,296 million tokens per day for a single evaluator. The full calculation is in the collapsible section.
Prices change fast and vary by contract, so this guide gives none: use your provider’s dated price list for cost_per_GB_month and cost_per_token. Compare both items over the same period. Storage is billed per month while the judge consumes every day (about 39 billion tokens over 30 days in this example). A single per-token rate is also a simplification, since output tokens are usually billed higher than input tokens.
For technical teams: the cost model and the example calculation
total_obs_cost = storage_cost + processing_cost + judge_cost
stored_GB = RPS * avg_payload_KB * 86400 * retention_days / 1024 / 1024storage_cost = stored_GB * cost_per_GB_month (per month)processing_cost = collector_cost + ingestion_costjudge_cost = RPS * 86400 * sample_rate * tokens_per_judge * cost_per_token (per day)Example at 100 RPS, 4 KB average payload, 30-day hot retention: stored_GB = 100 * 4 * 86400 * 30 / 1024 / 1024 = 988.8 GB, about 1 TB kept in hot storage storage_cost = 988.8 * cost_per_GB_month (per month)
Example judge at 100 RPS, 10 percent sample, 1500 tokens per judge: judgments = 100 * 86400 * 0.10 = 864,000 judgments per day judge tokens = 864,000 * 1500 = 1,296 million tokens per day for one evaluator judge_cost = 1,296,000,000 * cost_per_token (per day)Cost drivers, in order of importance
Section titled “Cost drivers, in order of importance”The first driver is payload size. Prompts, completions and retrieved contexts can each weigh tens of kilobytes and storage volume grows in proportion to traffic.
The second is LLM-as-judge evaluation, which consumes tokens at every judgment. A trivial evaluator running on every trace can double total token usage.
The third comes from high-cardinality attributes such as user_id, request_id or session_id. Attached to metrics rather than traces, they inflate metrics storage (section 7.3).
Mitigation strategies
Section titled “Mitigation strategies”- Compress and tier old trace storage: hot for seven days, warm for thirty, cold archive beyond.
- Use small models for online evaluation and keep large models for periodic audit batches.
- Restrict high-cardinality attributes to traces and keep metrics low-cardinality through aggregation.
- Sample LLM-as-judge runs rather than evaluating every trace.
- Tier evaluators by cost and run the expensive ones on a small sample plus flagged traces.
7.3 Cardinality management
Section titled “7.3 Cardinality management”A metric is stored like a table where each row is one combination of its label values. With one model and three features, you have three rows. Add the user ID and you get one row per user per feature: hundreds of thousands. That number of combinations is called cardinality. It is the single largest scaling risk in an AI observability platform, because AI workloads naturally have high-cardinality dimensions: prompt versions, model versions, retriever index versions, tenant IDs, user IDs.
Rules of thumb
Section titled “Rules of thumb”Attributes with thousands of distinct values per day belong on traces, not on metrics. Metrics dimensions must stay bounded: model_name (tens of values), feature (tens), tenant (hundreds at most). Per-user metrics are an anti-pattern: query traces by user_id instead.
Changing backends does not solve the problem. Some backends (VictoriaMetrics, Grafana Mimir, Thanos, or managed offerings) claim better handling of high cardinality than Prometheus alone; these are vendor statements, to be measured on your workload. The real answer remains to fix the metrics design.
7.4 PII and data sovereignty
Section titled “7.4 PII and data sovereignty”Prompts and completions often contain personal data, like the IBAN in our scenario. Three controls are recommended, all the more so in regulated contexts: redact before storing, control access, limit retention.
Redaction before storage
Section titled “Redaction before storage”Redacting means blacking out sensitive passages before filing, as you would on an official document. There are three techniques, from the simplest to the most expensive.
Regular expressions catch known patterns: email, phone, IBAN, credit card, social security numbers. The Collector’s redaction processor does this (blocked values masked or hashed, non-allowed keys deleted).
Named entity recognition (NER) catches names of people, organizations and places, which patterns cannot describe. The Collector does not do this natively. It takes an external service, for example a personal data detection engine called by the application or by an intermediate component, or a custom processor.
An LLM can finally scrub what depends on context, where regex and NER miss. Same constraint: it takes an external component, with its cost and latency.
Run regex redaction in the collector, before export. Place NER or LLM processing upstream of storage. Never in the storage backend, once the data has been written.
Access control
Section titled “Access control”The minimum is role-based access control (RBAC) on trace queries: each person sees what their role allows. Where the backend supports it, add field-level control: attribute keys stay readable by all, but prompt and completion bodies require elevated access. Finally, keep an audit log of all trace access, retained per regulatory requirement.
Retention policy
Section titled “Retention policy”Retention is aligned with the storage limitation principle (GDPR Article 5(1)(e)) and with sectoral regulations. Plan distinct retention for trace metadata (longer) and trace bodies (shorter). Automate purges and have them verified by audit.
7.5 Security
Section titled “7.5 Security”LLM observability opens an attack surface that classical APM (application performance monitoring) does not face. The observability platform itself becomes a vault holding a copy of the conversations.
Threats
Section titled “Threats”Stored prompts may contain secrets pasted by mistake: API keys, authentication tokens, passwords. Stored completions may contain training data the model memorized and gave back. Evaluation pipelines calling external LLM judges may send sensitive data out to third-party providers. Trace query interfaces show the full content of past interactions to anyone with access. Finally, a prompt injection can target the judge model itself and skew evaluator scores.
Mitigations
Section titled “Mitigations”- Scrub credential patterns before storage, alongside personal data.
- Deny outbound LLM judge calls by default, with an explicit allowlist of permitted providers.
- Control network egress from the collector to judge endpoints.
- Keep an audit log of all trace access and review it periodically.
- Encrypt trace storage, at rest and in transit.
- Harden judge prompts against injection: sanitize the input fields, use delimiters, never put the rated content in the system prompt.
7.6 Organizational ownership
Section titled “7.6 Organizational ownership”A platform with no owner degrades quickly. I recommend a simple split. The platform team owns the stack: collectors, backends, dashboards, alert routing. The ML team owns the evaluators, quality thresholds and the regression set. The security team owns redaction rules, access policies and audit review.
All three teams jointly run the weekly review of failed and low-score traces. In my view it is the most useful recurring forum and the only one that closes the loop.
The full RACI matrix (including product, compliance and quality on-call) is in the GenAI method, Part VII, which is the reference; the AI Act and regulatory evidence are in Part VI.
To start this week, check two things: where your data gets redacted before storage and who can read prompt content today.
Next: 8. Versioning, replay and experimentation.
Revised on 2 October 2026: prices removed, tail sampling limits (asynchronous evaluations and user feedback), memory sizing, NER and LLM redaction outside the Collector, GDPR Article 5, NIS2 and DORA location requirements corrected.
Revised on 4 October 2026: section 7.2 designated as the reference on the cost of observability, with a link to the method for the cost of non-observability; section 7.6 summarized in three lines with a link to the method’s RACI.
Revised on 25 October 2026: opening on an illustrative scenario, sampling, cardinality and redaction explained with images, cost model and processor parameters moved into collapsible sections, lists written as prose.