Skip to content

Dataset File Manifests and SHA-256 Checksums: Verify an Export Before Review

A byte-level manifest makes an export easier to check before technical, rights, or privacy review. Record what the package says it contains, then verify the exact bytes without turning an integrity check into a licensing conclusion.

A dataset file manifest is a small record of what an export contains. A useful row names one relative file path, its byte size, its media type, and a digest calculated from the exact bytes delivered. A reviewer can then check that the expected files arrived and that a file was not changed between packaging and review. SHA-256 is a practical digest choice for this integrity check. Python’s standard hashlib documentation exposes SHA-256 and accepts bytes as input. Neither source says that a digest proves ownership, consent, suitability for a model, or legal clearance.

Updated

On this page

Decide what the manifest means

The source dates in this guide identify documentation checks on 22 October 2026, not original publication dates. Follow each citation for the standard or documentation edition being discussed.

Before hashing anything, write the boundary of the package. A manifest can describe a folder, an archive, or each distribution inside an export. Do not mix these meanings in one file. These are proposed editorial fields, not a claim that Rightmark currently creates or verifies them.

For a checksum comparison to establish integrity against an expected package, trust or independently authenticate the manifest first. Record who supplied it and how the expected digest was obtained. If both arrive through the same untrusted channel, both the files and their manifest can be changed together. A match then establishes consistency with that manifest, not authentic origin or protection against deliberate replacement.

Decide what the manifest means (table)
Manifest fieldPractical meaningReview question
relative_pathStable path inside the packageIs the path safe to display and unambiguous inside this package?
size_bytesNumber of bytes in the delivered fileDoes the recorded size match the bytes that were checked?
sha256Digest of the delivered bytesDoes recomputing the digest produce the recorded value?
media_typeDeclared format, such as text/csv or application/jsonDoes the parser and file content agree with the declaration?
roleWhy the file is present, such as data, schema, documentation, or evidenceCan a reviewer tell whether a missing file changes the review scope?
version_idThe release identifier chosen by the ownerDoes this package belong to the intended release?
generated_atTime the manifest was written, with an offset or UTC formCan the manifest be compared with the export record?

Avoid absolute workstation paths, credentials, access tokens, private URLs, personal names, and raw records. A relative path is enough for a package-local check. If a filename itself contains personal or confidential information, use a controlled internal identifier and keep the mapping outside the public worksheet.

Hash the bytes before you transform them

Hash the file as delivered, opened in binary mode. Do not decode a CSV, normalize line endings, remove a byte-order mark, unzip and rezip, or rewrite JSON before calculating the package digest. Each of those actions can change bytes even when a human sees equivalent content. Python’s byte-oriented hashlib interface is the basis for this local example Python Software Foundation, (opens in a new tab).

Local byte-level example

Hash the bytes before you transform them code example, Python
from hashlib import sha256
from pathlib import Path


def sha256_file(path: Path) -> str:
    digest = sha256()
    with path.open("rb") as stream:
        for block in iter(lambda: stream.read(1024 * 1024), b""):
            digest.update(block)
    return digest.hexdigest()

The block size is an implementation choice. The important properties are that the stream is binary and that every byte is fed to the same digest in order. Record the algorithm name beside the value. A bare hexadecimal string is harder to interpret later.

If you create a normalized derivative for analysis, give it its own manifest row and digest. Do not replace the original row with the derivative’s hash. The original export and the derivative answer different questions.

Verify paths, sizes, and digests

Verification should fail closed. A checker should report a missing file, an unexpected file, a size mismatch, and a digest mismatch separately. A matching digest does not excuse a missing schema, a wrong media type, or a file placed under the wrong logical role.

Verify paths, sizes, and digests (table)
ResultWhat it tells youNext action
MissingThe package does not contain a manifest row’s fileStop and reconcile the package boundary.
UnexpectedA delivered file is not describedAdd it to the scope decision or quarantine it.
Size mismatchBytes differ in lengthPreserve both records and investigate packaging or transfer.
Digest mismatchThe checked bytes differ from the recorded bytesDo not silently regenerate the manifest. Identify which copy is authoritative.
Digest matches, parser failsBytes are unchanged but the declared format or content is unusableRecord the technical failure separately; integrity is not readability.
All checks matchThe described package is byte-consistentContinue to rights, privacy, schema, and sample review.

When an archive itself is the deliverable, decide whether the manifest hashes the archive, its unpacked members, or both. Hashing only an archive proves the archive bytes are stable. It does not make an individual member easy to locate or compare. Hashing only members does not prove the archive wrapper was transferred unchanged. The choice belongs in the package record.

Keep the integrity record separate from the rights record

A checksum can answer, “Are these the same bytes?” It cannot answer, “May this collection be licensed?” Keep title, authority, contributor terms, consent or notice evidence, privacy flags, exclusions, and intended use in their own review records. A file can pass a digest check while containing the wrong scope, personal data, confidential material, or a source whose terms do not permit the intended use.

Do not publish a real manifest that exposes private filenames or internal storage layout. Use a redacted or synthetic example when demonstrating the shape. Keep the actual package and any mapping from internal IDs in controlled storage.

Pair this byte-level check with the data provenance log and data card template when you need source, rights, exclusions, and scope evidence together. The broader export guidance covers export quality by tool; this guide owns the manifest boundary and digest question.

After the owner has a package boundary, a manifest, and evidence references, the Rightmark appraisal path may be the next preparation reference for an eligible owner. The appraisal path does not turn a checksum into approval, a quote, or a completed licensing review. Resolve rights, privacy, and contractual questions with the appropriate owner or counsel before sharing an archive.

Sources

  1. National Institute of Standards and Technology: FIPS PUB 180-4, Secure Hash Standard (opens in a new tab)
  2. Python Software Foundation: hashlib, Secure hashes and message digests (opens in a new tab)
  3. World Wide Web Consortium: Data Catalog Vocabulary, Version 3 (opens in a new tab)

Use the appraisal path only after the package boundary and evidence are clear.

The appraisal path does not validate checksums, approve a package, or create a licensing offer.