Technical
Integrity chain for audit trails: why a signed log is not an auditable log
For: engineers and SREs · architects · finance, risk and compliancePrerequisites: Basic knowledge of hashing, digital signatures and the OpenTelemetry Collector.
1. The initial misunderstanding
Section titled “1. The initial misunderstanding”In regulated environments, the conversation about log integrity risks stopping at one sentence: “the logs are signed”. The box is ticked, the topic is closed, the auditor leaves with a screenshot.
This reasoning contains a level error. A signature is a local property: it applies to one record taken in isolation. An audit asks a global question: what is missing from this log.
These two questions do not overlap. A log can be entirely signed, with every signature perfectly valid, and still be a fake. That is the subject of this article: what a signature really proves, what it does not prove and what you must build on top of it to get a log that a hostile third party can verify.
2. What a signature proves
Section titled “2. What a signature proves”A digital signature on a record r establishes two things:
- Authenticity of origin:
rwas produced by the holder of the private key. - Integrity of the record:
rhas not been modified since it was signed.
That is all. The signature says nothing about neighboring records, about order, about completeness, or about what should have been written and was not.
LOG SIGNED LINE BY LINE +----------------+ +----------------+ +----------------+ | r1 | sig(r1) | | r2 | sig(r2) | | r3 | sig(r3) | +----------------+ +----------------+ +----------------+ [OK] [OK] [OK] Each record is valid. No cryptographic link exists between them. The whole is not a verifiable entity.The signature guarantees the quality of the links. It does not guarantee that a chain exists.
3. The three blind spots
Section titled “3. The three blind spots”Three manipulations produce a log in which all signatures remain valid.
3.1 Selective deletion
Section titled “3.1 Selective deletion”The attacker removes the inconvenient records. The remaining records have not been touched: their signatures verify.
BEFORE AFTER DELETION r1 [OK] r1 [OK] r2 [OK] <- unauthorized access r3 [OK] r3 [OK] r5 [OK] r4 [OK] <- privilege escalation r5 [OK] Global check: 3/3 valid signatures Tool verdict: compliantThis is the simplest scenario for anyone who wants to cover their tracks and it is precisely the one against which per-record signing is useless.
3.2 Reordering
Section titled “3.2 Reordering”Chronology is a property of the whole. A timestamp included in the signed record does not help if the clock belongs to the compromised producer. Two swapped records remain two valid records.
3.3 Tail truncation
Section titled “3.3 Tail truncation”The attacker cuts off the end of the log, typically the moment when their presence becomes visible. Nothing in the remaining log signals that there was more.
Summary
Section titled “Summary”| Manipulation | Per-line signature | Hash chaining | Chaining + external anchoring |
|---|---|---|---|
| Modification of a record | Detected | Detected | Detected |
| Insertion of a record | Not detected if key compromised | Detected | Detected |
| Deletion in the middle | Not detected | Detected | Detected |
| Reordering | Not detected | Detected | Detected |
| Tail truncation | Not detected | Not detected | Detected |
| Complete rewrite by the key holder | Not detected | Not detected | Detected |
| Silent source (nothing is written) | Not detected | Not detected | Detected if heartbeat |
In my view, the last two rows are the ones most easily forgotten when presenting a logging solution.
4. Hash chaining
Section titled “4. Hash chaining”4.1 Mechanics
Section titled “4.1 Mechanics”Each record carries the digest of the chain state at the time it was written.
h(0) = H(chain_id) // seedh(n) = H( h(n-1) || chain_id || seq(n) || H(payload(n)) )Structure of a record:
+----------------------------------------------------------------+| chain_id : stream identifier (source, process, host) || seq : strictly monotonic integer, specific to chain || ts : producer timestamp (indicative, not probative) || prev_hash : h(n-1) || payload_hash : H(payload) || payload : the event itself |+----------------------------------------------------------------+| hash : h(n), computed over the fields above |+----------------------------------------------------------------+4.2 What chaining solves
Section titled “4.2 What chaining solves”CHAINING: DETECTING A DELETION r1 --h1--> r2 --h2--> r3 --h3--> r4 --h4--> r5 | | deletion of r3 v r1 --h1--> r2 --h2--> ??? --h3--> r4 r4.prev_hash = h3 digest recomputed up to r2 = h2 h2 != h3 -> BREAK DETECTED at position 4Verification is a simple linear replay, in O(n), with no secret: anyone who has the log can run it. This is the first interesting property. Verification becomes public.
4.3 The cost
Section titled “4.3 The cost”Chaining relies on hashing, not on asymmetric signatures. The values below are orders of magnitude, which vary widely with the processor and the library; measure them on your hardware with openssl speed sha256 ed25519:
| Operation | Order of magnitude | Practical throughput |
|---|---|---|
| SHA-256 on a 1 KB record | ~1 to 4 µs | from a few hundred MB/s to more than 1 GB/s per core, depending on whether the processor has SHA instructions |
| Ed25519 signature | ~20 to 60 µs | ~15 to 50 k signatures/s per core |
| Ed25519 verification | ~50 to 200 µs | ~5 to 20 k verifications/s per core, more expensive than signing |
The direct operational conclusion: you chain every record, you do not sign every record. With these orders of magnitude, signing line by line at 100,000 events per second would consume 2 to 6 cores (100,000 × 20 to 60 µs) and verifying those signatures 5 to 20 cores (100,000 × 50 to 200 µs), whereas chaining consumes 0.1 to 0.4 (100,000 × 1 to 4 µs) for a guarantee of sequence integrity. Signatures go on checkpoints, not on lines.
5. What chaining still does not solve
Section titled “5. What chaining still does not solve”In my view, this is where the most easily forgotten part lies.
5.1 Tail truncation
Section titled “5.1 Tail truncation”A truncated chain is still a consistent chain. A verifier who does not know the true head of the chain has no way of knowing that the chain continued.
COMPLETE CHAIN r1 -> r2 -> r3 -> r4 -> r5 -> r6 -> r7 head = h7CHAIN TRUNCATED BY THE ATTACKER r1 -> r2 -> r3 -> r4 head = h4 Linear verification: consistent end to end. Verdict: valid. Without an external reference to h7, the truncation is undetectable.5.2 Complete rewrite by the key holder
Section titled “5.2 Complete rewrite by the key holder”This is the structural point. If the audited entity holds the signing key and controls the storage, it can rebuild an entire alternative log: recompute the whole chain from the seed, re-sign the checkpoints, replace the file. The result is perfectly consistent.
In other words: integrity evidence produced and kept by the audited entity only has value for that entity. It proves the absence of accidents, not the absence of fraud. It protects against disk corruption and operational error, not against a hostile administrator or an attacker who has obtained the privileges of the producing process.
An HSM moves the problem without solving it: the key can no longer be exfiltrated, but the compromised process can still ask the HSM to sign whatever it wants. The HSM protects the key, not the truth.
5.3 Silence
Section titled “5.3 Silence”A source that stops emitting produces a perfectly valid log: the empty log is consistent. The absence of events is indistinguishable from the absence of a source. This is a simple way around the controls: you do not forge the log, you switch off the producer.
6. External anchoring: the tipping point
Section titled “6. External anchoring: the tipping point”The only way out is to move a piece of information outside the audited party’s sphere of control, periodically and irreversibly.
6.1 The signed and anchored checkpoint
Section titled “6.1 The signed and anchored checkpoint”+---------------------------------------------------------------+| CHECKPOINT || chain_id || seq_from, seq_to (coverage) || head_hash = h(seq_to) (chain state) || count (number of records covered) || signature (producer key) || tsa_token (qualified timestamp, RFC 3161) |+---------------------------------------------------------------+The checkpoint is published to a third party, at a short and fixed interval. From then on, the audited party can no longer rewrite the past before the last anchor without the divergence becoming visible.
Overview, from local chaining to anchoring outside the audited party’s sphere:
flowchart TB
subgraph AUD["Audited party's sphere"]
G["Seed<br/>h0 = H(chain_id)"] --> RN["r1 to rN<br/>h1 to hN"]
RN --> CP["Signed checkpoint<br/>head_hash = hN"]
end
subgraph EXT["Outside its control"]
TSA["Qualified timestamping<br/>RFC 3161"]
W["Third-party store<br/>WORM"]
end
CP -->|"32-byte digest"| TSA
TSA -->|"tsa_token"| CP
CP -->|"append only deposit"| W
6.2 Anchoring sequence
Section titled “6.2 Anchoring sequence”sequenceDiagram
autonumber
participant P as Producer (audited)
participant T as Qualified timestamping authority (TSA)
participant W as Third-party store / WORM
participant A as Auditor
P->>P: chains records (hashing)
P->>P: computes head_hash every N lines or T seconds
P->>T: timestamp request on head_hash
T-->>P: signed timestamp token
P->>W: deposits checkpoint (append only, immutable)
Note over W: The audited party can no longer modify what is deposited
A->>W: retrieves the sequence of checkpoints
A->>P: retrieves the raw log
A->>A: replays the chain, compares with anchored head_hash
A->>A: checks continuity of seq and coverage
6.3 The five properties obtained
Section titled “6.3 The five properties obtained”| Mechanism | Property obtained |
|---|---|
| Hash chaining | Internal integrity and order of the stream |
| Monotonic sequence number per source | Detection of gaps |
| Checkpoint anchored with a third party | Detection of truncation and rewriting |
| Qualified timestamping | Legally enforceable dating, independent of the audited party’s clock |
| Sealed periodic heartbeat | Detection of silence |
The heartbeat deserves a line: a chain must emit a record even when nothing happens. “Nothing to report” is information. Without it, switching off the collector becomes the simplest and most discreet way to erase.
6.4 Forward security
Section titled “6.4 Forward security”Against an attacker who obtains the key at time t and wants to rewrite what came before, the countermeasure is forward security: the key evolves at each period (k(n+1) = KDF(k(n))) and the previous key is destroyed. An attacker who captures k(n) cannot rebuild k(n-1) and therefore cannot re-sign the past. They control the future, never what came before. This family of constructions has been documented since Schneier and Kelsey’s work on secure audit logs and remains, in my view, little used in production.
6.5 Scaling up: Merkle tree
Section titled “6.5 Scaling up: Merkle tree”Linear replay becomes expensive on very large volumes (as an order of magnitude, from hundreds of millions of records, depending on hardware and how often you verify). The suitable structure is the Merkle tree, with two compact proofs:
ROOT (anchored) / \ N01 N23 / \ / \ H(r1) H(r2) H(r3) H(r4) | | | | r1 r2 r3 r4 Inclusion proof of r3 : {H(r4), N01} -> O(log n) Consistency proof C1->C2 : the tree has only grown, no past element was modifiedThe consistency proof is the decisive property: it shows that between two checkpoints, the log has only been appended to, never rewritten. This is the model of transparency logs, proven at very large scale by Certificate Transparency (RFC 9162).
7. Where to place chaining in an OTLP pipeline
Section titled “7. Where to place chaining in an OTLP pipeline”This is the concrete engineering question and the answer is counter-intuitive.
7.1 The three possible placements
Section titled “7.1 The three possible placements”flowchart LR
A["Application<br/>OTel SDK"] -->|OTLP| B["Collector<br/>receiver"]
B --> C["processors<br/>batch, filter, transform"]
C --> D["exporter"]
D -->|OTLP| E["Backend<br/>storage"]
E --> F["Offline<br/>verifier"]
P1["Option 1<br/>chain in the application"] -.-> A
P2["Option 2<br/>chain in the exporter"] -.-> D
P3["Option 3<br/>chain at backend ingestion"] -.-> E
Option 1, in the application. Rejected. Chaining becomes a dependency of business code, to be reimplemented in every language and maintained by every team. Coverage risks being partial, and therefore worthless: a partially chained log offers no overall guarantee.
Option 3, at backend ingestion. Rejected. The whole path between the producer and the backend stays out of scope. You seal the data after the only segment where it could be manipulated and ask the auditor to trust the backend, that is, the audited party.
Option 2, in the exporter. Selected. It is the last point where a single writer holds an authoritative sequence for a given source, before the data leaves the trust domain. A single Collector usually receives several sources: the exporter then keeps one chain per source (for example per service.instance.id) and not one chain shared by everything it receives. This is what reconciles this choice with the rule in the next section: the chain is computed in the pipeline, but it belongs to the source.
7.2 The Collector’s pitfalls
Section titled “7.2 The Collector’s pitfalls”The OpenTelemetry Collector was not designed to preserve strict ordering. Four behaviors break a naive chain:
| Behavior | Effect on the chain | Countermeasure |
|---|---|---|
batch processor, or exporter batching (sending_queue.batch): grouping | Order break | Chain after grouping, in the exporter |
Exporter send queue and retries (sending_queue, retry_on_failure): at least once delivery, several parallel consumers (num_consumers, 10 by default) | Duplicates, send order not guaranteed | Idempotent verification on (chain_id, seq); num_consumers: 1 if send order matters |
| Several Collector instances | Competing chains | One chain per source, never a global chain |
| Restart, buffer loss | Legitimate break | Seal and break record |
The queue, retry and batching options cited are those of the exporterhelper, which replaced the old queued_retry processor. Exporter batching is disabled by default (batch: {} to enable it) and is tending to replace the batch processor.
The rule that follows from all this fits in one sentence: the chain belongs to the source, not to the pipeline. Any attempt to build a single global chain across a distributed infrastructure that at best guarantees at least once delivery will produce permanent breaks that operations will end up disabling. The risk is a classic one: an integrity check that cries wolf so often that it gets switched off.
7.3 Handling legitimate breaks
Section titled “7.3 Handling legitimate breaks”A process restart is a normal break. It must be declared, not suffered.
END OF CHAIN START OF NEXT CHAIN+-------------------------+ +----------------------------+| type : SEAL | | type : GENESIS || chain_id : A | | chain_id : B || seq_to : 148,220 | ------> | prev_chain : A || head_hash : h(148220) | | prev_head : h(148220) || reason : shutdown | | seq : 0 || signature, tsa_token | | signature, tsa_token |+-------------------------+ +----------------------------+A chain that stops without a seal record is a warning signal, not an operational incident. This distinction is what separates a usable integrity system from a noise generator.
7.4 What the verifier does
Section titled “7.4 What the verifier does”The verifier is a standalone binary, with no access to the production infrastructure, that takes an exported log and a sequence of checkpoints as input. It produces a verdict and a list of anomalies. No tool is published with this article: what follows is an illustrative output of the verifier to be built, with values invented for the example.
$ verify --journal export.jsonl --checkpoints anchors/ --pubkeys keys/chain_id=svc-payment-01 records : 148,220 chaining : OK checkpoint coverage : 100% (612 checkpoints) qualified timestamps : 612/612 valid seq continuity : OK final seal : OKchain_id=svc-auth-03 records : 91,004 chaining : OK checkpoint coverage : 98.7% ANOMALY : seq 88,401 -> 88,460, 58 records missing ANOMALY : last checkpoint 14:02:11, last record 14:19:40 -> 17 min not anchored, possible truncation ANOMALY : no seal recordVERDICT : 1 compliant chain, 1 non-compliant chainWho checks what: each check performed by the verifier covers one family of manipulations and none is enough on its own.
flowchart LR V["Offline verifier<br/>exported log,<br/>anchored checkpoints,<br/>public keys"] V --> V1["Chain replay"] --> D1["Modification, insertion,<br/>deletion, reordering"] V --> V2["seq continuity"] --> D2["Gaps"] V --> V3["head_hash compared<br/>with anchors"] --> D3["Truncation, rewriting"] V --> V4["Timestamp tokens"] --> D4["Legally enforceable dating"] V --> V5["Heartbeat and seal"] --> D5["Silence, undeclared<br/>stop"]
This level of output is the deliverable to aim for. A green console is not evidence.
8. The regulatory framework, without marketing
Section titled “8. The regulatory framework, without marketing”The texts almost never mandate a named cryptographic mechanism. They mandate properties and it is up to the architect to choose the construction that delivers them.
| Text | Relevant requirement | Technical translation |
|---|---|---|
| DORA (EU 2022/2554) | Traceability of operations, protection and detection, ability to reconstruct the events that led to an incident | Complete, ordered, non-rewritten, legally enforceable log |
| NIS2 | Logging, chain of evidence for incident notification | Proof of completeness, reliable dating |
| PCI DSS, requirement 10 | Protection of logs against modification, tamper detection | Chaining plus integrity check |
| ISO 27001 A.8.15 | Logs protected against tampering and unauthorized access | Integrity and confidentiality |
| eIDAS (EU 910/2014), Article 41, paragraph 2 | A qualified electronic time stamp enjoys a presumption of the accuracy of the date and time and of the integrity of the associated data | One of the levers that gives a legal presumption under European law |
| eIDAS, Article 35, paragraph 2 | A qualified electronic seal enjoys a presumption of integrity of the data and of correctness of their origin | Seal checkpoints with a qualified seal |
| eIDAS, Article 25, paragraph 2 | A qualified electronic signature has the legal effect of a handwritten signature | Useful for an act signed by a person, less so for a machine stream |
| eIDAS 2 (EU 2024/1183), Article 45k, paragraph 2 (presumption) and Article 45l (requirements) | The data in a qualified electronic ledger enjoy a presumption of uniqueness and authenticity, of the accuracy of their date and time and of their sequential chronological ordering within the ledger; Article 45l sets the requirements such a ledger must meet, including that any later change to the data is immediately detectable | Deposit checkpoints in a qualified ledger |
The qualified services of eIDAS are, in my view, the most actionable and least exploited lever. Anchoring checkpoints on a qualified timestamp, sealing them with a qualified seal or depositing them in a qualified electronic ledger turns a technical property into evidence. The texts are on EUR-Lex: Regulation (EU) No 910/2014, consolidated version and Regulation (EU) 2024/1183. The cost is marginal: you timestamp a 32-byte digest every few seconds, not every line.
Two misconceptions to dismiss along the way:
- “WORM is enough.” Immutable storage protects against modification after writing. It protects neither against what was never written, nor against a compromised producer writing a fake into immutable storage. A lie carved in stone is still a lie.
- “The SIEM guarantees integrity.” The SIEM is downstream. It inherits the quality of what it is sent. It cannot certify what it never received.
9. The verifiability test
Section titled “9. The verifiability test”A single question distinguishes a signed log from an auditable log:
Can a third party that does not trust me, with its own tools, without my infrastructure and without my cooperation, detect a deletion, a truncation or a rewrite?
The resulting checklist:
- Each record carries the digest of the previous one, in a chain specific to its source.
- Sequence numbers are strictly monotonic and gaps are detectable.
- A sealed heartbeat is emitted even when there are no events.
- A signed checkpoint is published at a short interval, with a third party outside the audited party’s control.
- Checkpoints carry a qualified timestamp.
- The signing key evolves and earlier key material is destroyed.
- Chain breaks are declared by a seal record.
- The verifier is a standalone binary, runnable offline, whose output is an anomaly report and not an indicator light.
- An adversarial verification exercise is run periodically: you deliberately delete a record and measure whether the system sees it.
If the last point has never been carried out, the system has never been tested. It has been deployed.
10. Conclusion
Section titled “10. Conclusion”The signature answers the question “is this record authentic”. Nobody asks that question during an audit.
The question asked is “is this log complete”. It is global in nature and it calls for three distinct, stacked constructions: chaining to link records together, external anchoring to take away the audited party’s power to rewrite, qualified timestamping to provide a legally enforceable date.
Without anchoring, you get integrity evidence whose value is zero for the only person who needs it: the one who does not trust us.
The cost of this architecture is low and the calculation is simple: hashing on every line, a signature every few seconds, a timestamp on 32 bytes. In my view, cost is not what explains its absence. It is the fact that the “signed logs” box can be ticked without it.
Revised on 2 October 2026: eIDAS legal presumptions completed (qualified seal, qualified electronic ledgers of eIDAS 2), Collector options updated (sending_queue and retry_on_failure instead of queued_retry), verifier output presented as illustrative, SHA-256 and Ed25519 orders of magnitude widened and core calculation redone, eIDAS 2 reference clarified (Article 45k, paragraph 2 and Article 45l), Certificate Transparency sourced.