On this page
Define the span and the claim
The source dates shown in this guide identify documentation checks on 22 October 2026, while RFC 5646 retains its September 2009 publication month. Start with the tagged unit: document, page, paragraph, sentence, speaker turn, or token span. Record whether the tag describes observed content, intended audience, interface locale, translation target, or a detector output. Those are different claims and should not share one field.
| Field | Record | Stop if |
|---|---|---|
| Span | Stable item and character or segment boundary | Document tag applied to mixed spans |
| Language | BCP 47 tag and source of assignment | Code copied from folder name |
| Script | Script subtag only when distinguishing | Script inferred from region |
| Region | Region context and evidence | Region treated as dialect |
| State | Observed, inferred, unknown, or mixed | Detector output called truth |
| Version | Tagging policy and registry snapshot | Policy is implicit |
RFC 5646 recommends using precise tags no more specific than justified. A short en tag may be safer than a guessed region or script. Keep the original tag, normalized form, and any review decision as separate fields.
Check syntax without overclaiming meaning
Validate the tag structure and the order of subtags, then ask whether each added subtag is supported. Script can distinguish written forms when that difference matters. Region can describe usage or locale context, but it is not automatic evidence about a speaker, author, or dialect. Extensions and private-use values need a documented application meaning.
| Tag case | Example fixture | Review question |
|---|---|---|
| Language only | en | Is extra specificity necessary? |
| Language and region | en-US | Does region describe content or audience? |
| Script | sr-Latn | Does script add useful distinction? |
| Multiple languages | mul | Are spans separately available? |
| Undetermined | und | Is evidence insufficient or absent? |
| Private use | x-fixture | Is private meaning documented? |
Unicode CLDR distinguishes BCP 47 language tags from CLDR locale identifiers and documents transformations such as hyphen to underscore for locale data. Keep protocol tags and internal locale keys in separate fields so a normalization step does not erase the source form.
Represent mixed, code-switched, and unknown content
A multilingual document can include quotations, names, code, transliteration, and code-switching. If a corpus stores only one document tag, note what span-level evidence was lost. Use an explicit unknown or undetermined state when the source does not support a language claim. Do not turn a detector confidence score into a BCP 47 tag without retaining the detector version, candidate set, and review policy.
- Record the smallest stable span available.
- Preserve the producer tag and detector output separately.
- Use script and region only when evidence supports them.
- Mark mixed or undetermined content explicitly.
- Keep code, symbols, and non-linguistic content distinct.
- Record policy and registry versions for normalization.
Use a synthetic language-tag worksheet
Synthetic corpus tag rows
item | span | producer_tag | detector_state | review_state
fixture-1 | whole-document | en-US | matches language only | keep en-US with region evidence unknown
fixture-2 | paragraph-2 | sr-Latn | script observed | keep
fixture-3 | sentence-4 | und | mixed content | split span or keep und
fixture-4 | code-block | zxx | no linguistic content | keep separate from prose
fixture-5 | quote-span | de | detector unavailable | source tag only; confidence unknown| Test | Expected result | Hold condition |
|---|---|---|
| Region | Region claim separated from language | Region treated as dialect |
| Script | Script retained only when observed | Script inferred from filename |
| Mixed span | Span-level split or explicit mixed state | One tag hides code-switching |
| Unknown | Undetermined state preserved | Guessed language inserted |
| Normalization | Source and normalized forms kept | Tag replaced by locale key |
Every item and state in this fixture is synthetic. Add a tag with an extension, a private-use tag with no policy, and a truncated tag. The review should request evidence or preserve uncertainty rather than inventing language metadata.
Keep language metadata separate from corpus conclusions
Language-tag review can show what metadata says and where granularity or evidence is missing. It cannot establish author identity, audience, translation quality, cultural meaning, privacy, or suitability for a downstream task. Preserve source tags, span boundaries, registry versions, and review states.
Pair this guide with text encoding row checks for byte and decoding evidence and Unicode normalization and identifier comparison for string representation. After a supported non-health scope, authority to describe the collection, and a controlled multilingual packet are established, the qualified appraisal request path may be a preparation step. It does not identify languages, translate content, create a quote, match a buyer, or grant a license.
Takeaways
- Tag a defined span and state what the tag claims.
- Use BCP 47 specificity only when the evidence supports it.
- Keep script, region, mixed, unknown, and non-linguistic states distinct.
- Preserve source tags beside normalized locale keys.
- Do not treat a valid tag as proof of correct language labeling.