On this page
Keep physical types and logical meaning separate
Parquet uses physical storage types plus logical annotations. The Apache Parquet, (opens in a new tab) gives BYTE_ARRAY with a STRING annotation as one example and defines DATE and TIMESTAMP forms. Retain both layers in the worksheet.
| Column fact | Synthetic observation | What remains to decide |
|---|---|---|
| Physical type | INT32 | Count, code, date carrier, or another meaning? |
| Logical annotation | DATE | Does the coverage rule match a calendar date? |
| Timestamp parameters | isAdjustedToUTC plus unit | Instant or local date-time field? |
| Nullability or definition levels | Optional field with nulls | Unavailable, not applicable, or export failure? |
| Nested path | profile.contact.region | Is it in the published schema? |
For timestamps, the specification distinguishes an adjusted-to-UTC instant from a local timestamp that needs more time-zone context. That is a format interpretation, not business meaning. Preserve producer documentation beside the annotation.
Do not infer semantic units from a physical type. An INT64 can be an identifier, count, epoch value, or encoded measure. A BYTE_ARRAY can be text, binary content, or an annotated logical type. If the annotation is missing or unknown, record the uncertainty rather than silently casting it.
Treat directory partitions as a separate dataset contract
A Parquet file can be readable while its dataset root is partitioned inconsistently. Apache Arrow, (opens in a new tab) documents several schemes: directory partitioning maps path segments to fields in a declared order, Hive partitioning uses key-value directories such as year=2025/month=04, and filename partitioning places values in a declared filename order.
These are dataset-layer conventions, not a replacement for the Parquet footer. Record the scheme and whether field types were declared or inferred. A different partition schema can interpret the same path differently. Arrow also notes that filters may use partition information or Parquet statistics to reduce a scan, but a filtered scan is a query result, not a coverage certificate.
Synthetic partition tree
synthetic-export/
year=2025/month=01/part-000.parquet
year=2025/month=02/part-000.parquet
year=2026/month=01/part-000.parquet| Path observation | Safe state | Next action |
|---|---|---|
| Every leaf has declared year and month keys | Partition shape documented | Compare values and coverage with the manifest |
| A key is missing from one branch | Partial contract | Hold until fallback and scope are explicit |
| Key values have mixed types, such as 01 and January | Ambiguous values | Define a canonical type and preserve the path |
| Path disagrees with an in-file column | Possible stale key | Test the producer rule on an approved sample |
| Directory contains unexpected files | Boundary unclear | Quarantine or classify before scanning |
| Empty partition exists | No row evidence | Record an empty path, not a populated slice |
Use row groups and page evidence carefully
The Apache Parquet, (opens in a new tab) describes column chunks as pages written back to back. Page headers describe compression and encoding, and an optional column index can help skip pages. The Apache Parquet, (opens in a new tab) describes page-level lower and upper bounds for that use.
| Finding | What it can support | What it cannot establish |
|---|---|---|
| Row-group and row counts are available | A bounded comparison with manifest coverage | That every row is accurate or authorized |
| Column statistics have bounds or null counts | A targeted range or nullability investigation | That statistics cover a complete dataset |
| Page index is present | A possible selective scan under a compatible reader | That every reader prunes the same way |
| Statistics are absent | An explicit unavailable-evidence state | That the column has no values or no quality |
| Files have different schemas | A schema-evolution or partition review | That one schema safely represents all files |
Name the file, row group, column path, evidence type, reader version, and unresolved question. If a row is needed to confirm a metadata finding, use an approved synthetic or redacted artifact; do not copy a private row into a public worksheet.
Sources
- Apache Parquet: File Format (opens in a new tab)
- Apache Parquet: Logical Types (opens in a new tab)
- Apache Arrow: pyarrow.dataset.partitioning (opens in a new tab)
- Apache Parquet: Metadata (opens in a new tab)
- Apache Parquet: Column Chunks (opens in a new tab)
- Apache Parquet: Page Index (opens in a new tab)
- Apache Parquet: Encryption (opens in a new tab)