Exact vs near-duplicate dataset review: a two-pass method

The Python documentation citations use the date checked, 22 October 2026, not an original publication date. The government framework and NIST publication retain their dated source editions.

Write the unit before choosing a matching rule. Possible units include a file, table row, document, message, media asset, event, or version. A duplicate decision for files does not automatically apply to rows inside those files. Then define the permitted action: keep one copy, retain both as versions, merge metadata, exclude a repeated export wrapper, or hold the pair for an owner decision.

The UK Government, 3 Dec 2020 (opens in a new tab) treats uniqueness as one dimension rather than a universal similarity threshold. Choose the threshold and action in a review contract, with a reason and an owner.

For files, compute a digest over the exact bytes and retain a manifest row for each file. The digest identifies the bytes that were read; it does not prove that two files have the same provenance, rights, or intended meaning. Keep the source reference, declared version, format, size, and digest together.

For rows, define a canonical serialization before hashing. Choose field order, representation of missing and null values, date-time form, numeric formatting, and treatment of whitespace. If field order is not part of meaning, sort by the dictionary rather than a parser’s incidental order. If field order is meaningful, preserve it and state that choice.

Typed canonical serialization with synthetic collision checks

import hashlib
import json
import math

def typed_value(value):
    if value is None:
        return ["null"]
    if isinstance(value, bool):
        return ["boolean", value]
    if isinstance(value, int):
        return ["integer", str(value)]
    if isinstance(value, float):
        if not math.isfinite(value):
            raise ValueError("non-finite floats need an explicit policy")
        return ["number", value.hex()]
    if isinstance(value, str):
        return ["string", value]
    raise TypeError("adapt non-JSON values explicitly before hashing")

def exact_key(row, columns):
    tokens = []
    for column in columns:
        if column not in row:
            tokens.append([column, "missing"])
        else:
            tokens.append([column, typed_value(row[column])])
    payload = json.dumps(
        tokens, ensure_ascii=False, separators=(",", ":")
    ).encode("utf-8")
    return hashlib.sha256(payload).hexdigest()

# Synthetic collision checks only. No production records are used.
assert exact_key({}, ["a"]) != exact_key({"a": None}, ["a"])
assert exact_key({"a": None}, ["a"]) != exact_key({"a": ""}, ["a"])
assert exact_key({"a": "x|y", "b": "z"}, ["a", "b"]) != exact_key(
    {"a": "x", "b": "y|z"}, ["a", "b"]
)

The Python Software Foundation documentation, 22 Oct 2026 (opens in a new tab) documentation supports the SHA-256 interface used here. The tagged row serialization is an editorial rule, not a library default. Record the column order, missing and null tags, scalar type tags, numeric encoding, JSON encoding, and selected columns. Adapt dates, decimals, bytes, arrays, and objects with explicit rules before hashing. Never hash a secret as a substitute for a privacy review.

Near-duplicate review begins by choosing safe normalization rules. Unicode normalization can reconcile different code-point representations, but compatibility normalization can also erase distinctions that matter. The Python Software Foundation documentation, 22 Oct 2026 (opens in a new tab) documentation describes NFC, NFD, NFKC, and NFKD as different forms. Pick one only when the field definition supports it, and record the form.

• Trim surrounding whitespace only when the field definition treats it as presentation noise.
• Standardize line endings when line-ending choice is not part of the content identity.
• Compare case for a case-insensitive code only when the codebook says case is not meaningful.
• Do not lower-case opaque identifiers, remove punctuation from legal text, or discard timestamps merely because values look similar.
• Normalize a comparison copy and retain the source representation for the decision record.

Use blocking fields to create candidate pairs before a similarity calculation. A block might use a declared source ID, document family, date window, language, or file group. If no safe block exists, record that the candidate set may be incomplete. There is no honest universal threshold for “near enough”; test the chosen rule against synthetic fixtures with obvious matches, legitimate versions, and false positives.

Use explicit dispositions

• candidate: a rule found a possible match that needs review.
• confirmed_duplicate: the owner-approved rule supports the disposition.
• confirmed_distinct: the records remain separate for a documented semantic reason.
• version: the content overlaps but version history is meaningful.
• unknown: available evidence cannot support a conclusion.

Do not publish a pair list that allows a reader to infer membership in a sensitive collection. NIST’s National Institute of Standards and Technology, Sep 2023 (opens in a new tab) treats de-identification as a risk and governance activity. Run exact checks over approved metadata or controlled storage first. Use synthetic fixtures for matching logic. Request a redacted artifact only when its purpose, audience, approval, and deletion path are documented.

General information, not legal advice.

If a supported non-health collection has a clear owner and scope, the qualified next step may be the appraisal route. It is a request for review, not automatic acceptance, a quote, a buyer match, a legal opinion, or a license. This guide does not promise that a deduplication result changes an appraisal outcome.

Takeaways

• Define the unit, comparison rule, and permitted action before matching.
• Use exact hashes to find identity, then review near matches with documented normalization.
• Preserve versions, repeated events, conflicts, and unknowns.
• Keep pair outputs neutral and do not expose sensitive content to prove a match.