On this page
Separate visual equality from string equality
Two identifiers can render the same way while carrying different code point sequences. A composed character and a base character followed by a combining mark are the standard synthetic test. A reviewer who prints only the value may see no difference; a byte comparison, hash, or exact join can still distinguish them. Start by preserving the source text and inspecting code points or a safe escaped representation.
The source dates shown in this guide identify documentation checks on 22 October 2026, not original publication dates. Record the Unicode version, runtime, and normalization library used by the review. A future Unicode or runtime update may change which characters are permitted or how an application treats them, so the review should be reproducible rather than implied by a display screenshot.
| Comparison result | What it tells you | What it does not tell you |
|---|---|---|
| Visual match | The rendered text looks alike | The strings are byte or code point equal |
| Exact match | The source sequences are equal | The values are canonically equivalent |
| NFC match | The chosen canonical forms are equal | The identifiers have the same business owner |
| NFKC match | Compatibility distinctions were treated as equal | That loss is acceptable for every consumer |
| Collision | Different source strings share a comparison key | Which record should win |
Choose a normalization policy per field
Do not normalize every string in a dataset with one blind replacement. A display label, an exact external key, a search key, and a protocol identifier have different contracts. For each field, decide whether the original text is authoritative, whether a comparison key is derived, whether case folding is allowed, and what happens when two originals produce the same key.
| Policy field | Worksheet choice | Decision to document |
|---|---|---|
original | preserve | Which value is shown and exported? |
normal_form | NFC, NFD, NFKC, NFKD, or none | Which equivalences are intentional? |
case_policy | exact, lower, or a specified fold | Is case part of identity? |
comparison_key | derived_and_not_authoritative | Can the key be regenerated? |
collision_action | hold_for_review | Can distinct originals map together? |
unicode_version | record_runtime_version | Can another reviewer reproduce it? |
UAX #15 explains that compatibility normalization can remove distinctions that may matter to identifiers. Use NFKC only when the field owner has decided that those distinctions are not identity-bearing. UAX #31 offers identifier guidance, but a standard default still needs to be reconciled with the producer contract and the receiving system Unicode Consortium, (opens in a new tab) Unicode Consortium, (opens in a new tab).
Derive a comparison key and retain the original
A comparison key is useful only when it is a declared derivative. Store it beside the original, include the chosen form and version in the review record, and never overwrite the source text just to make a join pass. The key can support an exact, canonical, or compatibility comparison, but it cannot decide a business collision by itself.
import unicodedata
left = "cafe\u0301"
right = "caf\u00e9"
def comparison_key(value: str) -> str:
return unicodedata.normalize("NFC", value)
assert left != right
assert comparison_key(left) == comparison_key(right)
review = {
"original_left": left,
"original_right": right,
"comparison_left": comparison_key(left),
"comparison_right": comparison_key(right),
"form": "NFC",
"case_policy": "exact",
"unicode_version": unicodedata.unidata_version,
"key_equal": True,
"action": "compare_owner_and_context",
}
Collision worksheet
- Capture both original strings in an escaped or code point-safe form.
- Record the form, case rule, Unicode version, and comparison key.
- Check whether the producer says the originals are the same identifier.
- Hold the pair when different originals collapse to one key without an owner rule.
- Publish the comparison result without erasing the source evidence.
Test exports, joins, and canonicalization separately
Run the policy at every boundary where strings are parsed, stored, indexed, sorted, hashed, or exported. A source may preserve an original while a warehouse key normalizes it, or a JSON serializer may preserve a string while a downstream comparison silently applies a different case rule. Test the same fixture through each system and compare both the original and derived key.
| Boundary | Test | Evidence to retain |
|---|---|---|
| Input parse | Inspect code points before normalization | Escaped original and parser version |
| Storage | Read back original and comparison key | Round-trip result |
| Join | Compare exact and policy-derived keys | Matched, unmatched, or collision state |
| Sort or index | Check ordering and lookup behavior | Key policy and locale assumptions |
| JSON canonicalization | Confirm strings are unchanged unless policy runs first | Pre and post text |
RFC 8785 canonicalizes JSON representation but preserves Unicode strings as-is. It can make serialized bytes deterministic without making canonically equivalent identifiers equal. Run the identifier policy before a comparison, and treat canonical serialization as a separate transport check Internet Engineering Task Force, (opens in a new tab).
Take a qualified next step
If the owner can describe the collection and its handling authority, the next step may be the appraisal route. That route is a request for review, not automatic acceptance, a quote, a buyer match, a legal opinion, or a license. Use synthetic Unicode pairs to test the policy first; do not send private names merely to demonstrate a normalization collision.
Takeaways
- Preserve original text before deriving any comparison key.
- Choose normalization and case rules by identifier role.
- Use compatibility normalization only when its information loss is intended.
- Detect collisions and hold ambiguous pairs for an owner decision.
- Keep Unicode policy separate from JSON canonicalization and visual display.