On this page
Name the IPC boundary before reading values
Start with the artifact boundary: an Arrow IPC stream is consumed in order, while an IPC file provides a footer and block locations for random access. A file copied without its footer, a stream cut after a batch, or a concatenation of two streams can look like a reader error even when the underlying batches were individually valid. Preserve the original filename, byte length, first and last marker evidence, and the capture time as metadata rather than guessing from a reader exception.
The Apache Arrow, (opens in a new tab) documentation describes separate stream and file readers. That distinction is operational: a stream has one sequential message flow and a file has a footer that indexes blocks. Do not call a stream a file because its extension is arrow or feather, and do not use a file footer as evidence that a stream was complete.
| Boundary field | Record | Stop when |
|---|---|---|
| container_kind | stream, file, or unknown | The kind is inferred only from a filename |
| capture_scope | single export, shard, or concatenation | The source scope is not stated |
| start_evidence | magic, first message, or unavailable | The first bytes were stripped |
| end_evidence | end marker, footer, or unavailable | The copy ended before closure |
| format_version | declared IPC version and source | Only a library version is known |
| byte_extent | length and immutable artifact id | Length changed during review |
Compare schema and record batch shape
A record batch contains arrays that share a logical row count. Record the field order, field names, nested children, nullability, metadata, and the length of every batch. A schema that is textually similar can still change meaning when a timestamp timezone, decimal scale, list child, extension name, or dictionary index type changes. Preserve the full schema message or a redacted structural digest; a short column list is not enough to replay the interpretation.
The Apache Arrow, (opens in a new tab) format specification describes arrays with a type, buffers, length, null count, and optional dictionaries. That gives a useful review vocabulary, not a permission to inspect private values. Compare structure first, and keep unknown application semantics visible when the schema does not state units, code lists, or provenance.
| Shape check | Evidence to keep | Failure meaning |
|---|---|---|
| field order | ordered field paths | A consumer may bind by position |
| field type | type parameters and extension name | Same name does not mean same type |
| batch length | row count per batch | Batch alignment is uncertain |
| null count | declared count and observed state | Null bitmap may be incomplete |
| dictionary state | id, values, and replacement rule | Labels may resolve differently |
| metadata | key names, values, and source | Metadata may describe another export |
Treat dictionaries and metadata as state, not decoration
Dictionary encoding introduces a second piece of state: indexes in an array point to dictionary values. Record when a dictionary is introduced, which id it serves, whether a later dictionary replaces it, and which batches depend on that state. A value that prints as the same label in two readers can still come from different dictionary versions. If the dictionary is missing, do not silently replace indexes with a best guess.
Keep application metadata separate from format metadata. A key such as source, units, or release may be useful context, but its presence does not prove that the value was generated by that source or that the description is still current. Store metadata as observed, cite its source, and mark unverified claims as questions for the owner.
Dictionary and metadata questions
- Which schema message establishes the dictionary index type and value type?
- Does every dictionary id used by a batch have an observed definition before that use?
- Is a dictionary replacement allowed by the chosen IPC container and version?
- Which metadata keys describe the batch, and which describe the export process?
- Can a reviewer distinguish absent metadata from an intentionally empty value?
Use a small synthetic message ledger
Before opening sensitive batches, test the review worksheet with invented field names and short values. The fixture below is a ledger, not an IPC parser. It exercises the order that a reviewer must account for and makes a missing dictionary, changed schema, or truncated end visible without copying a customer row. Keep the fixture in the review notes and keep the real payload outside a public worksheet.
Compare the expected message sequence with what the artifact manifest says. If the manifest records only a file size and a reader status, the result is incomplete. A reader can stop at a valid boundary while the capture is still missing a later batch or a footer.
| Observed state | Disposition | Next evidence |
|---|---|---|
| Schema first, all fields stable | continue | Compare batch lengths and null counts |
| Dictionary id used before definition | hold | Locate the preceding dictionary message |
| Field metadata changed mid-stream | hold | Version the schema boundary |
| Last batch has no end marker | incomplete | Obtain the original stream or capture note |
| Footer indexes all file blocks | continue | Check block offsets and batch inventory |
Synthetic IPC message ledger
artifact: ipc-fixture-01
container: stream
sequence: schema -> dictionary(3) -> record_batch(3) -> record_batch(2) -> end
schema: order_id:string, status:dictionary<int8>, amount:decimal(10,2)
dictionary(3): 0=queued, 1=sent, 2=failed
check: every batch has 3 fields and dictionary id 3 before status valuesMake failures about evidence, not reader optimism
A successful open answers only whether one implementation accepted the bytes. It does not answer whether the capture is complete, whether the schema matches another export, or whether a field has the intended business definition. Report a narrow disposition such as schema mismatch, dictionary state unresolved, incomplete stream, footer unavailable, or application metadata unverified.
When comparing Arrow with Parquet, keep the question distinct. Parquet review focuses on file metadata, row groups, and partition evidence; IPC review focuses on message sequence, schema and dictionary state, record-batch boundaries, and stream versus file framing. Link to the Parquet metadata guide when a pipeline crosses formats, and record the conversion boundary rather than treating the two containers as interchangeable.
| Finding | Disposition | Do not infer |
|---|---|---|
| Schema differs in field type or timezone | hold comparison | That one file is corrupt |
| Batch lengths differ but schema matches | investigate capture scope | Rows are duplicated |
| Dictionary state is incomplete | hold dependent labels | The labels are obvious |
| Stream end marker absent | mark incomplete | The last batch is final |
| Metadata is present but unverified | retain as context | The claimed source is proven |
Close with a qualified preparation step
Once the IPC boundary, schema, dictionary state, batch inventory, and unresolved questions are recorded, keep the packet at structural evidence level. Do not paste raw batch values into a public request. Use the data provenance log for capture and transformation history and the data card template for collection context.
For a supported non-health collection, with authority to describe the artifact and a controlled handling boundary, the qualified appraisal path can be a next preparation step. It does not parse IPC files, certify a schema, create a quote, match a buyer, issue a legal opinion, or grant a license. Arrow format evidence also does not prove that the underlying values are accurate or permitted for a proposed use.
Takeaways
- Identify stream versus file before interpreting a batch.
- Keep schema, dictionary, record-batch, and metadata state separate.
- Use a synthetic message ledger before exposing real values.
- Report a narrow evidence disposition instead of treating reader success as completeness.
- Carry format-conversion boundaries into any Parquet or table comparison.