On this page
Validate each WARC record boundary first
Record a neutral file ID, digest, byte offset, WARC version, WARC-Type, Content-Length, record ID, WARC-Date, and the end offset calculated from the block length. The International Internet Preservation Consortium, (opens in a new tab) specification says every record has a header and content block, and every record carries Content-Length. Do not scan for the next WARC line and call that a boundary when the declared block length disagrees.
The source dates in this guide use 22 October 2026 as retrieval evidence for documentation without a verified publication date. They are not publication claims. Where a verified publication date is available, it is recorded in the source ledger.
| Boundary field | Record | Stop if |
|---|---|---|
| Record header | Version and named fields | Header terminator is missing |
| WARC-Type | Defined type or extension type | Reader maps an unknown type to response |
| Content-Length | Declared octets and observed span | Block extends beyond file |
| Record ID | URI and uniqueness within the review | A record cannot be linked |
| WARC-Date | UTC value and precision | Local display replaces source value |
| End marker | Required line endings and next offset | Reader skips unexplained bytes |
Keep compressed file framing and decompressed record framing separate. A digest or byte offset should say which layer it covers. If a parser repairs a header, retain the raw evidence and identify the repair rather than replacing the record.
Separate response block, HTTP message, and payload
In a response record, the content block can contain HTTP headers followed by a data object. The WARC specification distinguishes the block from the payload, and its Content-Type describes the archived HTTP message rather than simply copying the HTTP Content-Type header. Record the WARC block span, HTTP header terminator, HTTP message type, payload start, payload length if present, and any mismatch with an inner Content-Length.
| Layer | Record | Cannot infer |
|---|---|---|
| WARC block | Outer Content-Length and byte span | That all bytes are payload |
| HTTP headers | Status, headers, and line ending | That headers are trustworthy application data |
| HTTP body | Payload start and observed length | That body length matches outer block |
| Identified type | WARC-Identified-Payload-Type | That MIME type proves content |
| Compression | Outer file and inner transfer state | That one codec applies at all layers |
| Truncation | WARC-Truncated and source note | That a replay is complete |
Synthetic response ledger
{"record_id":"urn:fixture:response-01","type":"response","target_uri":"https://example.invalid/fixture","warc_date":"fixture-UTC","block_length":"fixture-octets","http_status":"fixture-status","payload_offset":"fixture-offset","payload_digest":"fixture-digest","truncated":"not-recorded"}Link a capture event without collapsing record types
The International Internet Preservation Consortium, (opens in a new tab) specification uses WARC-Concurrent-To to associate records from one capture event, while WARC-Refers-To links a metadata, revisit, or conversion record to the record it describes or transforms. Build an evidence graph with direction and relationship type. A request, response, revisit, metadata, and conversion record are not interchangeable snapshots.
| Relation | Record | Open question |
|---|---|---|
| Concurrent-To | Peer record IDs from one capture | Is the association present on both sides? |
| Refers-To | Record ID being described or transformed | Is the referenced record available? |
| Target URI | Original URI and normalized comparison key | Was a query or fragment changed? |
| Revisit | Target and prior record references | Which payload is actually stored? |
| Continuation | Origin ID, segment number, total length | Were all segments reassembled? |
| Conversion | Source record and derived payload | What transformation occurred? |
Keep dates, digests, and replay claims bounded
WARC-Date records when capture began and is expressed as a UTC timestamp. WARC-Block-Digest can cover the full record block, while WARC-Payload-Digest covers the payload. Record algorithm labels, digest scope, source record, and comparison result. A matching payload digest supports byte identity for that scope; it does not prove that the captured response was truthful, complete, or authorized.
| Evidence | Supports | Does not prove |
|---|---|---|
| WARC-Date | Capture timing as recorded | When the source content was originally created |
| Block digest | A digest for the full block | The payload alone is unchanged |
| Payload digest | A digest for the payload scope | Headers and metadata match |
| Truncation field | A recorded truncation state | Unrecorded truncation did not happen |
| Replay output | A rendering produced by a tool | Original response semantics |
| Capture graph | Records can be associated | Every related record was captured |
Stop when archive evidence is incomplete
Stop conditions
- Stop if record version, type, ID, date, Content-Length, or end offset is missing or contradictory.
- Stop if the response block, HTTP headers, and payload boundary cannot be separated.
- Stop if capture associations rely on target URI text without record IDs or relation types.
- Stop if continuation segments, truncation, revisit references, or conversion sources are incomplete.
- Stop if digest scope or algorithm is unknown, or a replay is treated as original payload evidence.
- Stop before opening real payloads when target URIs, headers, or bodies expose private records.
Pair this worksheet with the data provenance log for capture and transformation history and the data card template for collection context. A valid WARC boundary does not prove content completeness, ownership, privacy clearance, or permission.
For a supported non-health collection, after the owner has authority to describe the scope and a controlled metadata packet, the qualified appraisal request path may be a preparation step. It does not crawl, replay, authenticate, assess web content, create a quote, match a buyer, or grant a license.