On this page
Define which order you mean
Before testing an event package, write down the order that matters. Keep source_position, event_time, processing_time, package_sequence, and entity_key as separate fields. If a source provides no stable position, write UNAVAILABLE; do not create one from the row number unless that row number is explicitly part of the source contract. Sources checked on 22 October 2026. Dates in the Sources list are retrieval dates, not claimed publication dates.
| Order dimension | Meaning | Common mistake |
|---|---|---|
| Source order | Position assigned by the source log or transaction | Treating wall-clock time as source sequence |
| Transaction order | Changes grouped or ordered within one source transaction | Assuming separate transactions share one total order |
| Entity order | Changes for one stable key | Applying an update before the create for that key |
| Partition order | Position within one delivery partition | Assuming total order across partitions |
| Batch order | Order in the exported file or package | Treating file order as business chronology |
| Business order | Domain sequence such as effective date or revision | Overwriting a correction because its event time is older |
Delivery order and business order can both matter, but they answer different questions. State which one controls application, which one supports audit, and which one remains unavailable.
Use stable identities for replay checks
An idempotency key answers: “If this event is applied twice, how will the consumer recognize the same operation?” It may be a source stream identifier plus partition and offset, a source transaction and row sequence, or a source-assigned event ID. The correct combination depends on the source. A business entity key alone is often insufficient because two legitimate updates to the same entity must remain distinct.
| Field | Review question |
|---|---|
event_id | Is it source-assigned, deterministic, or unavailable? |
entity_key | Which entity or relationship does the event affect? |
operation | Create, update, delete, snapshot, correction, or other? |
source_position | Unique within a partition, table, stream, or source? |
sequence | Can it detect an older event arriving after a newer one? |
payload_digest | Audit aid only, not an identity claim |
apply_policy | Upsert, ignore replay, queue, compensate, or hold? |
Do not use a content hash as a universal event identity. The same content can be emitted twice legitimately, and two different events can carry the same state. The exact and near-duplicate review owns content identity questions; this guide owns change-event consumer behavior.
Know what delivery systems guarantee
The Apache Software Foundation, (opens in a new tab) documentation describes each Kafka partition as a totally ordered log with a consumer position. Key-based partitioning can route an entity’s records to one partition, giving a per-partition boundary rather than total order across partitions. Do not turn that partition-local statement into a cross-stream total-order claim.
The Apache Software Foundation, (opens in a new tab) documentation says idempotence can suppress producer-retry duplicates under the required configuration. That concerns records in the Kafka log; it does not make an external update idempotent or stop a restarted consumer from applying an event twice. Kafka documents at-least-once behavior when processing precedes position saving.
The Debezium project, (opens in a new tab) documentation gives a bounded example with transaction and log positions in source metadata and a recorded WAL position for restart while a replication slot remains available. A generic export must document which positions it actually has. Sources checked on 22 October 2026 are retrieval labels here.
Test a synthetic replay and reorder fixture
Fictional event positions and delivery attempts
# Synthetic fixture only. No production records are used.
source position entity operation business time delivery attempt
stream-A:41 order-7 create 10:00 first
stream-A:42 order-7 update 10:02 first
stream-A:42 order-7 update 10:02 retry
stream-A:43 order-7 delete 10:03 first
stream-B:9 order-8 update 09:58 arrives late- Decide whether a repeated source identity is ignored, applied safely, or sent to a review queue.
- Check whether a delete remains after a consumer restart or a re-read from an earlier position.
- Compare events only within their source stream unless cross-stream order is defined.
- Treat an older business time as a late event or correction only under a documented rule.
- Keep delete markers separate from null payloads when the source semantics require it.
| Finding | Safe review disposition |
|---|---|
| Same source identity seen again | Verify idempotent application or record a replay |
| Same entity, newer source sequence | Apply under the source ordering rule |
| Same entity, older sequence arrives later | Hold, compensate, or apply under a correction rule |
| Same timestamp, different source positions | Use source position; do not invent tie order |
| No stable event identity | Mark replay detection unavailable |
| Different partitions or streams | Avoid a total-order claim unless the source defines one |
Reconcile delivery order with business state
Compare both the event ledger and the resulting state. Count events by operation and source partition, record the earliest and latest known positions, and compare expected entity transitions with the synthetic or owner-approved source contract. A matching message count does not prove that the right entities changed. A repeated message does not necessarily mean the source data is duplicated; it may be a consumer retry.
| Evidence field | Purpose |
|---|---|
package_id | Connects the review to the export package |
stream and partition | Limits the ordering claim |
positions | Records the source boundary actually observed |
events_read | Counts deliveries, not unique business changes |
unique_identities | Counts source identities after replay review |
replayed_identities | Makes duplicate delivery visible |
held_events | Preserves unresolved order or identity questions |
state_check | Records the resulting synthetic or approved-state comparison |
Keep raw messages, names, speech, medical content, and private identifiers out of public worksheets. A synthetic fixture demonstrates ordering and replay without exposing a real collection. The dataset version and change-log guide owns release identity, and the data provenance log owns source and transformation history.
Keep exactly-once language qualified
“Exactly once” can refer to publication, delivery, processing, or final side effects. Those are different claims. Kafka documents transactional processing for specific Kafka Streams and transactional producer and consumer arrangements, while external systems may need their own coordination. Use precise language such as “producer duplicate suppression configured,” “consumer replay detected,” or “destination side effect not independently verified.” If the evidence does not establish the layer, write UNAVAILABLE.
Event-order checks also do not establish ownership, permission to share, privacy clearance, accuracy, buyer demand, pricing, or licensing readiness. Keep those questions in the owner’s data card, provenance record, and applicable review process. After the handling boundary is documented, the appraisal route can be a qualified preparation reference for an eligible, supported non-health collection. It is a request for review, not automatic acceptance, a quote, a buyer match, a legal opinion, or a license. Do not upload sensitive raw event streams merely to demonstrate replay behavior.
General information, not legal advice.
Takeaways
- Name the ordering scope before comparing events.
- Use a source-aware event identity; an entity key or timestamp alone is rarely enough.
- Test duplicate delivery, late events, partition boundaries, and consumer restart with synthetic fixtures.
- Keep delivery guarantees, destination side effects, rights, privacy, and licensing decisions separate.