Check Text Encoding and Malformed Rows Before You Review an Export

The source dates in this guide identify documentation checks on 22 October 2026, not original publication dates. Follow each citation for the standard or documentation edition being discussed.

Start with an immutable copy or a digest-linked package record. Perform diagnostics on a working copy, but never overwrite the source file to make an error disappear. Record the expected encoding, delimiter, quote character, escape behavior, header policy, and expected columns before running the parser.

RFC 4180 documents a common CSV shape, including records separated by line breaks and fields separated by commas, with quoting for commas, line breaks, and quote characters RFC Editor, Oct 2005 (opens in a new tab). It is informational and does not make every CSV file conforming. Treat a producer’s declared dialect as part of the export contract rather than guessing from a sample.

Use binary mode when preserving or hashing the source. Use text mode only after choosing an explicit encoding and newline behavior. This example fails on invalid UTF-8 and keeps the diagnostic separate from any replacement-based derivative Python Software Foundation, 22 Oct 2026 (opens in a new tab).

Strict UTF-8 read

from pathlib import Path


def read_utf8_strict(path: Path) -> str:
    with path.open("r", encoding="utf-8", errors="strict", newline="") as stream:
        return stream.read()

If this raises UnicodeDecodeError, record the byte position and stop the normal review path until the owner identifies the encoding or supplies a corrected export. Do not immediately switch to errors="replace". Python documents replace as inserting a replacement character, which can change a name, identifier, or free-text value Python Software Foundation, 22 Oct 2026 (opens in a new tab). If a replacement derivative is required for exploratory work, keep it under a new path, give it its own digest, and label it as transformed.

A byte-order mark is also a fact to record, not a reason to silently rewrite the file. Python exposes BOM constants for codecs Python Software Foundation, 22 Oct 2026 (opens in a new tab). If a producer intentionally writes a BOM, document whether the parser accepts it and whether the original bytes remain available.

Python’s csv module recommends opening files with newline='' so the parser can handle line endings itself. Its Dialect.strict setting raises csv.Error on bad input. csv.Sniffer is heuristic, so an inferred delimiter should be a hypothesis to confirm, not a production contract Python Software Foundation, 22 Oct 2026 (opens in a new tab).

Strict CSV inspection

import csv
from pathlib import Path


def inspect_csv(
    path: Path, expected_columns: int, *, dialect: str | csv.Dialect
) -> list[dict[str, object]]:
    if expected_columns < 1:
        raise ValueError("expected_columns must be positive")
    if isinstance(dialect, str):
        dialect = csv.get_dialect(dialect)
    issues: list[dict[str, object]] = []
    reader = None
    try:
        with path.open("r", encoding="utf-8", errors="strict", newline="") as stream:
            reader = csv.reader(stream, dialect=dialect, strict=True)
            header = next(reader, None)
            if header is None:
                return [{"kind": "empty_file", "row": None}]
            if len(header) != expected_columns:
                issues.append({"kind": "header_width", "row": 1, "actual": len(header)})
            for row_number, row in enumerate(reader, start=2):
                if len(row) != expected_columns:
                    issues.append({
                        "kind": "row_width",
                        "row": row_number,
                        "actual": len(row),
                    })
    except UnicodeDecodeError as error:
        issues.append({
            "kind": "decode_error",
            "decoder_buffer_start": error.start,
            "decoder_buffer_end": error.end,
            "physical_line": reader.line_num if reader is not None else None,
        })
    except csv.Error:
        issues.append({
            "kind": "csv_error",
            "physical_line": reader.line_num if reader is not None else None,
        })
    return issues

Pass the documented dialect explicitly: dialect="excel-tab" selects a tab-delimited export. For a semicolon export, pass an instance of a csv.Dialect subclass with delimiter ; and the producer’s quote, escape, and double-quote rules. The parser validates that dialect rather than guessing. The decoding offsets above are relative to the decoding buffer, not absolute file offsets; physical_line is the parser’s last consumed line and can be unavailable or precede a buffered decode failure. Neither diagnostic copies source bytes or exception text.

These are illustrative local-fixture examples, not hardened parsers for hostile input. The strict reader loads the file into memory, and the CSV inspector does not cap retained issues or set an explicit field-size policy. Before using either on large or untrusted input, set byte, field-size, issue-count, memory, and execution-time limits and record early termination. The inspector reports empty files, width mismatches, decoding failures, and CSV parser failures; it does not repair rows, check header uniqueness, or validate required fields. Logical row numbers can differ from physical lines when a quoted field contains a newline. A matching field count does not establish valid dates, unique identifiers, or schema compliance.

For a large file, a streaming pass can record row numbers and counts without loading all content into memory. If you sample instead of scanning the whole export, state the sample rule and do not describe the result as a full-file check.

Do not reduce the result to a single pass or fail label. The reason determines the next action.

Encoding and row checks answer whether a technical parser can read the delivered representation. They do not answer whether the source was collected lawfully, whether personal data is present, whether a contributor permitted the proposed use, or whether an intended buyer’s requirements are satisfied. Keep the quality report linked to, but separate from, the provenance, permissions, exclusions, and privacy records.

Avoid publishing raw error rows when they may contain personal, confidential, or identifying text. A safe report can include the relative file path, row number, error class, byte offset when available, and a redacted sample or stable internal reference. Restrict the mapping from that reference to the actual row to the people who need it.

Pair this check with the data provenance log and the broader export guidance. The provenance log owns source, rights, processing, and exclusion evidence; the export guidance owns tool-specific export quality; this guide owns encoding and row-structure diagnostics.

Once the owner has a preserved source, an explicit parser contract, and a redacted quality report, the Rightmark appraisal path may be a next preparation reference for an eligible owner. It does not turn a parser pass into a quote, approval, or completed licensing review. Resolve rights, privacy, and contractual questions with the appropriate owner or counsel before sharing an export.