On this page
Define what counts as disagreement
The source dates shown in this guide identify documentation checks on 22 October 2026, not original publication dates. Follow each citation for the metric or research discussion being used. First define the annotation unit and comparison: exact label, span boundary, geometry, timestamp, ranking, multilabel set, or free-text rationale. A disagreement record needs the task version and the two outputs, not just a boolean mismatch.
| Disagreement layer | Example observation | Question to ask |
|---|---|---|
| Unit | Annotators selected different spans | Did they see the same token or region boundary? |
| Label | Same span, different class | Are classes mutually exclusive and defined? |
| Multiplicity | One versus several labels | Is multilabel annotation allowed? |
| Boundary | Intervals overlap but start or end differs | What tolerance rule was specified? |
| Abstention | One annotator used unknown | Is abstention an allowed state? |
| Rationale | Same label, different explanation | Is rationale part of the target or context? |
Do not label every mismatch as annotator error. A missing example, a new domain pattern, a broken instruction, or a genuinely ambiguous item may need a guideline change rather than a forced consensus. Preserve the original outputs before any adjudication.
Choose agreement evidence that matches the task
Agreement summaries are conditional measurements. The scikit-learn, (opens in a new tab) documentation describes a kappa statistic for categorical assignments and its chance-agreement setup. It does not make kappa suitable for every task. Span boundaries, multilabel sets, ordinal labels, missingness, and heavily imbalanced categories need a documented comparison rule and may need more than one summary.
| Task shape | Evidence to retain | Do not infer |
|---|---|---|
| Single categorical label | Contingency table and metric definition | Universal label quality |
| Ordinal label | Distance or weighting rule | That all disagreements have equal cost |
| Multilabel set | Per-label and set-level comparison | That one aggregate hides missing labels |
| Span or box | Overlap rule and boundary examples | That a near boundary is an exact match |
| Timestamp interval | Start, end, tolerance, and overlap policy | That durations are equally comparable |
| Open rationale | Theme or rubric comparison | A categorical score without coding rules |
The ACL paper ACL Anthology, (opens in a new tab) highlights metric selection and transparent reporting. Use its lesson as a reporting boundary: name the unit, label set, annotator count or role description when available, comparison rule, missing-data rule, and unresolved disagreement set. If those inputs are unavailable, say so instead of backfilling a number.
Record resolution without erasing the disagreement
Adjudication is a new activity, not a replacement for the source annotations. Record who or what resolved the item, which guideline version was consulted, whether the decision changed the codebook, and whether the item remains ambiguous. If a gold label is created, retain both the original outputs and the reason the gold label was selected.
- Freeze the original annotation records and task version.
- Classify the disagreement before opening a resolution discussion.
- Check whether the guideline has a relevant example or abstain state.
- Record the resolution rationale and any codebook change separately.
- Revisit comparable unresolved items only under the new rule and version them.
Synthetic disagreement ledger
item_id | output_a | output_b | class | resolution
synthetic-01 | invoice | receipt | boundary-meaning | guideline-example-needed
synthetic-02 | active | unknown | missing-context | abstain-retained
synthetic-03 | cat | dog | label-definition | adjudicated-v2
synthetic-04 | span-A | span-B | boundary-tolerance | unresolvedUse a pattern matrix before publishing a summary
A useful review aggregates disagreement by cause and task slice, not just by annotator. Use synthetic groups such as document type, label pair, boundary length, language, or time window. Keep group definitions broad enough that a public report does not reveal an individual item or rare source pattern.
| Pattern field | Example value | Follow-up |
|---|---|---|
| Label pair | invoice versus receipt | Compare definitions and examples |
| Boundary class | same object, edge differs | Specify tolerance and visual examples |
| Abstention rate | unknown used under missing context | Add an evidence-request rule |
| Guideline version | synthetic-v2 | Separate pre-change and post-change results |
| Resolution state | open, adjudicated, retained | Report unresolved items explicitly |
| Evidence status | source absent or synthetic only | Do not generalize beyond the fixture |
Keep disagreement evidence separate from downstream decisions
Disagreement review can show how a defined task behaved for a selected annotation set. It cannot establish accuracy, annotator fitness, privacy clearance, rights, demand, or suitability for a deployment. Keep provenance, privacy, rights, and owner decisions separate.
Pair this guide with ontology versioning for label history and missing-value and label consistency review for abstention states. After supported non-health scope, authority to describe the collection, and a controlled disagreement ledger are established, the qualified appraisal request path may be a next preparation step. It does not adjudicate labels, certify agreement, create a quote, match a buyer, issue a legal opinion, or grant a license.
Takeaways
- Define the annotation unit, label set, tolerance, and abstention rule before measuring disagreement.
- Choose evidence that matches categorical, ordinal, multilabel, span, box, or interval tasks.
- Retain original outputs when creating an adjudicated or gold record.
- Report patterns, guideline changes, and unresolved items instead of one universal score.
- Keep agreement evidence separate from accuracy, privacy, rights, and downstream-use decisions.