On this page
Define the independent unit before splitting rows
The source dates shown in this guide identify documentation checks on 22 October 2026, not original publication dates. Follow each citation for the split behavior being discussed. Begin with the prediction event and the unit expected to be new at evaluation time. That unit may be a row, person, document, source device, conversation, account, location, product, or time window. If it is unknown, mark independence as unavailable.
| Boundary question | Synthetic answer | Leakage risk |
|---|---|---|
| What is one prediction event? | One synthetic document | Several rows from one document can cross splits |
| What should be new? | A document family | Near-duplicates can share wording |
| Which fields are known at prediction time? | Metadata before event | Post-event fields may reveal target |
| What period is evaluated? | Later synthetic window | Random mixing may use future context |
| What transformations are fitted? | Vocabulary and normalization | Fitting on all rows can use test information |
| What is the grouping key? | Synthetic source_id | Missing or reused keys hide overlap |
The scikit-learn, (opens in a new tab) guidance says test data should not be used to make model choices and that transformations should be learned from training data only. Treat that as an information-flow question: record what the transformation saw, not only where the output row landed.
Trace direct and indirect information paths
Inspect more than duplicate row ids. Leakage can travel through a shared source, a derived feature, a normalization statistic, a label generated after the prediction event, a document revision, a session, or an evaluation decision fed back into training. Draw the path from source artifact to split assignment and mark each transformation and time boundary.
| Path | Observation to record | Hold condition |
|---|---|---|
| Entity overlap | Group key and cross-split counts | Group key missing or reused ambiguously |
| Exact duplicate | Stable digest or source identity | Deduplication rule unavailable |
| Near duplicate | Normalization and similarity rule | Comparison scope is unknown |
| Time travel | Event time and feature availability | Feature created after prediction |
| Fitted transform | Rows and partition used to fit | Test or future rows included |
| Target-derived field | Field lineage and generation time | Label or outcome leaks into input |
| Human review loop | Decision and feedback path | Evaluation influenced later labels |
If the dataset has groups, scikit-learn, (opens in a new tab) documents a splitter that keeps a group out of both train and test within a split. That implementation is not a universal answer: groups must represent the independent unit, and a missing or unstable group key can invalidate the assumption.
Match the split strategy to the deployment question
A random row split, group split, time split, and source-held-out split answer different questions. Record the intended deployment scenario and choose a strategy that preserves the relevant boundary. Do not select the strategy because it produces a more favorable result.
- Name the unit that must remain unseen.
- Freeze the split rule and seed or ordering evidence before fitting transformations.
- Apply learned preprocessing from the training partition to other partitions without refitting on them.
- Inspect duplicates, source ids, time intervals, and target-derived fields after splitting.
- Document what the selected split cannot simulate, such as a new source, later period, or unseen group.
Synthetic split ledger
record_id | group_id | event_time | split | feature_state | finding
synthetic-01 | source-A | t1 | train | known-before-event | allowed
synthetic-02 | source-A | t2 | test | known-before-event | group-overlap
synthetic-03 | source-B | t3 | test | fitted-on-all-rows | transform-leakage
synthetic-04 | source-C | t4 | test | post-event-field | time-leakageThe fixture is invented. A finding names an information path, not a model result. Retain the split manifest and transformation-fit scope so another reviewer can reproduce the boundary without seeing source records.
Use a split reconciliation worksheet
| Worksheet field | Example entry | Unknown state |
|---|---|---|
| Prediction event | Document classification at ingest | Event definition unavailable |
| Independent unit | Document family | Group key not supplied |
| Split rule | Group-held-out by source id | Rule or seed unavailable |
| Fit scope | Train partition only | Transformation history unavailable |
| Time boundary | Features before event time | Feature availability unknown |
| Overlap check | Digest and group comparison | Near-duplicate method unavailable |
| Disposition | Allowed, hold, or unresolved | Owner decision pending |
The scikit-learn, (opens in a new tab) documentation describes multiple cross-validation strategies and their assumptions. Use a strategy name only with its grouping, ordering, and preprocessing context. A split manifest can prove how rows were assigned; it cannot prove that the chosen unit matches a future deployment.
Keep split evidence separate from downstream decisions
Split review can explain whether selected records, groups, times, and transformations crossed a declared boundary. It cannot establish accuracy, privacy clearance, rights, demand, or model fitness. Keep provenance, privacy, rights, and owner decisions separate.
Pair this guide with exact and near-duplicate review for identity checks and the data provenance log for source and transformation history. After supported non-health scope, authority to describe the collection, and a controlled split packet are established, the qualified appraisal request path may be a next preparation step. It does not split data, detect leakage, benchmark a model, create a quote, match a buyer, issue a legal opinion, or grant a license.
Takeaways
- Define the independent unit and prediction-time information before assigning rows.
- Trace group, duplicate, time, transform, target, and feedback paths separately.
- Match random, group, time, or source-held-out splits to the deployment question.
- Preserve split manifests, transformation-fit scope, and unknown states.
- Keep leakage evidence separate from accuracy, privacy, rights, and model-fitness claims.