On this page
Define the watermark and boundary
A watermark is a source progress marker that an exporter can retain and use to define the next slice. It may be a log position, immutable sequence, version token, or timestamp paired with a stable tie-breaker. A local extraction time is not automatically a source watermark. Write the boundary contract before running a package.
| Field | Example meaning |
|---|---|
source_name | Database, API, file feed, or owner-defined source |
watermark_type | Log position, sequence, version, timestamp plus key, or unknown |
lower_bound | Prior committed watermark and whether it is inclusive |
upper_bound | Captured source cutoff and whether it is inclusive |
tie_breaker | Stable key used when many rows share a timestamp |
scope_predicate | Tables, partitions, status, and date limits included |
snapshot_or_consistency_ref | Source snapshot, transaction, or not available |
next_watermark | Value safe to use only after package validation |
late_change_policy | Overlap, correction queue, replay, or unresolved |
An inclusive lower bound paired with a stable key can make a boundary deterministic. An overlap can also be valid when the destination is replay-safe. The dangerous state is an undocumented mixture of strictly greater-than and greater-than-or-equal comparisons, local clock assumptions, and a cursor advanced before the package is verified. Sources checked on 22 October 2026. Dates in the Sources list are retrieval dates, not claimed publication dates.
Prefer source progress over guessed time
The PostgreSQL Global Development Group, (opens in a new tab) documentation describes an initial snapshot followed by changes sent continually, with source order within a subscription. The PostgreSQL Global Development Group, (opens in a new tab) documentation distinguishes initial synchronization from later DML operations and documents replica identity for updates and deletes. These are concrete source behaviors, not a universal exporter contract.
The Debezium project, (opens in a new tab) documentation is a separate implementation reference. It describes a consistent snapshot, WAL positions, committed row-level changes, and source metadata such as a log position, transaction identifier, snapshot state, and source timestamp. Use those fields only when the selected source actually emits them. Sources checked on 22 October 2026 are labeled as retrieval dates in this guide.
- If only a mutable timestamp exists, record its precision and update behavior.
- Use a tie-breaker, overlap, or source-provided cursor when rows can share timestamps.
- Record whether the cursor is source-owned, exporter-owned, or unavailable.
- Keep source event time separate from extraction time and package publication time.
Separate extraction from cursor advancement
| Run state | What it means |
|---|---|
planned | Bounds and scope written; no source claim yet |
reading | Source slice is being extracted |
packaged | Files, manifest, schema, and evidence references exist |
reconciled | Counts and boundary checks were reviewed |
committed | The next run may start from the recorded watermark |
held | A gap, overlap, schema change, or unresolved source state needs review |
Do not call a run complete because a file was written. Save the prior watermark, bounds, source cutoff, manifest reference, and reconciliation status together. The file manifest and checksum guide owns package bytes. The dataset version and change-log guide owns release identity. This guide owns the incremental boundary and cursor decision.
Reconcile the slice against the source contract
| Fixture case | Expected observation | Review question |
|---|---|---|
| New row after prior cursor | Included in the next slice | Was the lower bound interpreted as intended? |
| Two rows with the same timestamp | Tie-breaker or overlap handles both | Could one row be skipped? |
| Update after extraction began | Included, held, or deferred under a stated rule | Is the source snapshot consistent? |
| Late backfill with an older timestamp | Overlap, correction queue, or source sequence catches it | How are late changes found? |
| Delete or tombstone in the window | Recorded as a change, not inferred from absence | Does the package retain deletion evidence? |
| Schema change at the boundary | Held or versioned explicitly | Which parser and schema apply? |
Privacy-aware reconciliation record
# Synthetic fixture only. No production records are used.
run_id | source | lower_bound | upper_bound | key_tie_rule | rows_read |
package_id | rows_written | held_rows | source_ref | dispositionCompare source-side and package-side observations at the same declared grain. Useful checks include row counts by partition, distinct key counts, minimum and maximum watermark, held-row counts, and boundary keys. Do not compare counts across different scopes and call the difference loss. Explain an exclusion, late event, replay, correction, delete, or unresolved gap separately.
Handle overlap and replay deliberately
An overlap can be safer than a narrow timestamp boundary only when the load is replay-safe. Define the target operation: upsert by stable key and version, insert with an event identity, or hold duplicates for review. Do not silently discard a repeated row just because it appeared in two adjacent packages. It may be a retry, a correction, or the same business state observed under a new package.
- Test restart behavior with fictional keys and source positions.
- Retain source progress as evidence when the source offers a durable log position.
- Record whether a cursor was advanced before or after reconciliation.
- Mark restart guarantees unavailable when the source cannot provide the needed evidence.
The cited connector documentation describes resuming from a recorded WAL position while its replication slot remains available Debezium project, (opens in a new tab). That is connector-specific and is not a guarantee for other exporters. Do not generalize the example into a Rightmark product claim.
Keep rights separate and choose a qualified next step
A reconciled export says nothing about permission to disclose it. Keep source authority, collection purpose, exclusions, retention, and intended use in the data provenance log and owner review. A clean watermark does not establish a legal basis, buyer demand, price, or product eligibility.
After the owner has documented the source, scope, exclusions, and handling boundary, the appraisal route can be a qualified preparation reference for an eligible, supported non-health collection. It is a request for review, not automatic acceptance, a quote, a buyer match, a legal opinion, or a license. Do not upload sensitive raw records just to demonstrate cursor behavior.
General information, not legal advice.
Takeaways
- Define the source watermark, bounds, inclusivity, tie-breaker, and late-change rule before extracting.
- Advance the cursor only after package and reconciliation evidence are held together.
- Use overlap only with an explicit replay-safe target operation.
- Keep source progress, package identity, deletion evidence, privacy, and rights as separate questions.