Why a data profile should come before a private sample

An archive can be described accurately and still leave its relevance unresolved. A date range tells you when records were created. A system list tells you where they came from. Neither explains what the buyer needs those records to demonstrate. The first task in AI data sourcing is to connect the description to a decision.

My November request asked for parameters because the buyer's answer was still needed before selecting a set. An owner cannot answer a requirement that has only been inferred. If a buyer mentions an interest in long histories, does it need continuity across particular years, a minimum span or examples of an older workflow? Those are possible interpretations with different evidence needs. Asking the owner for more files does not resolve which interpretation the buyer intends.

The same is true of linked company records. Rightmark's research page describes connections among a request, the work it prompted and its result. For a particular sourcing request, the next question is which connection matters. A buyer might need tickets joined to their recorded resolutions. Another might want the discussion preceding a decision. A list of systems containing both kinds of records would not establish that either relationship survives in the available collection.

Put the buyer's condition beside the relevant owner fact, and give each a source. The buyer may have stated the condition in writing, or an intermediary may still be checking an interpretation. The owner may have measured the field, reported it from memory or left it unanswered. Those differences belong in the profile because they determine who can resolve the next question.

Rightmark's buyer request brief makes room for unknown values and asks for the unit and source behind a number. It also separates what is known, what needs evidence and what is unavailable for the request. That gives the owner a useful response even when it cannot fill every field.

For example, an owner can explain that the requested history is absent, while a different field has simply not been checked. Combining both under an incomplete-profile label would hide a decisive distinction. One answer may close this request. The other identifies work that could still make the decision possible.

Two questions have to stay separate: how much does this condition matter, and what do we know about meeting it? Rightmark's internal requirement record distinguishes hard filters from preferences, with evidence recorded separately as stated, inferred or unknown. Its rule for an unknown requirement is that it counts as neither a pass nor a fail. That is a useful discipline for reviewing a profile, not a claim that every buyer follows the same process.

A hard condition sets a boundary for the particular request. A preference affects how attractive a possible match is. Missing evidence tells you that a finding remains open. A single score can obscure those differences: a strong result on several preferences cannot supply a year of history that an essential condition requires.

Consider a fictional support archive (an example; the company and the requests are invented). All dates, languages, conditions and observations below are assumptions for this example. Assume an owner-held inventory confirms records from January 2023 through December 2024, no records from 2022 and English-language records only. The owner also reports that ticket IDs appear in a resolution table, but nobody has checked the coverage or meaning of those links. These observations establish no rights or permission to release records.

The archive stays the same across three independent requests:

Request A needs no attractive sample of 2023 tickets to settle the missing-year question. The proposed material cannot answer it. Request B leaves a possible match, even though another collection might better satisfy that preference. Request C needs evidence about a claimed relationship. Treating all three as failures would discard the distinction that tells the owner what to do next.

This is also why an unknown should be specific. If the buyer has not said whether Spanish is essential, the buyer needs to clarify the condition. If the owner does not know which languages its archive contains, the owner needs to establish that fact. The blank cell may look identical on a spreadsheet, but the unanswered questions belong to different people.

In a June 2022 Google interview, Mahima Pushkarna, then a senior interaction designer on the People + AI Research team, explained that “identifying what we don’t know about data is just as important as articulating what we do know.” Her point about dataset documentation applies here: an explicit unknown preserves a question that an unsupported answer would conceal.

Now change Request B so that Spanish becomes essential. The archive can be ruled out on that condition, but the reason is a changed request. Its records have not become worse. Keep the buyer's revised condition and its date beside the finding so that another reviewer can see why the decision changed. Otherwise, a later rejection can make an earlier, reasonable preference-based assessment look like an error.

Return to Request C in the invented example. A report that two tables contain ticket IDs is a starting point. Before preparing private content for inspection, the owner can check whether those IDs connect the tables within the requested period. The useful output is a bounded description of that check, including what was counted and what was left out.

For this example, a technical owner could prepare the following report. The fields are a proposed format; no check has been run and no results are supplied:

Link check for Request C

Scope: Tickets from January 2023 through December 2024 and the resolution records proposed for that same review.

Matching rule: State the ticket key used in each table and any transformation applied before matching.

Unit: Count distinct tickets, rather than rows created by joining the tables.

Coverage: Report distinct tickets in scope, tickets with at least one corresponding resolution, tickets with no match and tickets with multiple matches.

Limits: Record excluded records, missing keys and any ambiguity in what a match means.

The counting unit matters. Suppose a ticket can legitimately have several resolution records as work progresses. Counting the joined rows could make one ticket appear to supply several covered tickets. Counting distinct tickets with at least one match answers the narrower coverage question. Multiple matches then remain a separate finding to explain, rather than being silently inflated into coverage or automatically dismissed as corrupt data.

A failed match under one key also needs interpretation. The owner might discover that the resolution table uses a different documented identifier. That would call for correcting the matching rule. If no usable relationship can be established, the owner has evidence of a mismatch with Request C. The distinction prevents a technical choice made during measurement from quietly becoming the buyer's eligibility rule.

Even a structurally plausible result leaves another question. Does the linked resolution actually address that ticket? A matching identifier does not tell the reader whether the text records a result, an interim update or an unrelated note. This is where inspecting selected pairs can add evidence the count cannot supply. The next review can examine whether the records show the relationship the buyer needs.

Rightmark's sample-pack guidance starts by choosing the question before selecting records. It also calls for context about dates, formats and gaps, plus approval of the exact sample, recipient, purpose and transfer path. For Request C, that means selecting material for the relationship question and explaining the selection. A useful review of those pairs would still leave collection-wide coverage to the broader check.

Code provides a clear parallel. Rightmark's Repo Check can count test files, but it does not run tests, builds or installers. A test-file count alone therefore cannot establish a passed run. The tool can separately read an existing JUnit CI report and report outcome counts and whether its commit matches the checked repository. Those are reported CI outcomes, not tests executed by Repo Check. The evidence supports different conclusions depending on which field you are reading.

For the owner, this is a way to keep the next request proportionate. If the uncertainty concerns coverage, ask for the defined count. If it concerns the meaning of linked content, specify what the sample review must inspect. If the buyer cannot yet explain the relationship it needs, return to the buyer before asking an engineer to investigate an increasingly elaborate interpretation.

The strongest objection to profile-first sourcing is that a researcher may not know which characteristics matter until seeing examples. A form built around an established task can miss the feature that makes an unusual collection useful. Requiring every field before any exploration may send the owner to measure the wrong things.

There is a documented reason to resist fixed questions. In their November 2022 Google Research account, Mahima Pushkarna and Andrew Zaldivar, then Senior Developer Relations Engineer at Google Research, described a data governance group adapting documentation for an ML health-related initiative. The group used an activity to “rule out typical documentation questions, and identify atypical documentation needs specific to their data type”. Their example concerns documentation design. It challenges the assumption that a standard form already contains the right questions.

My case for an exploratory sample follows from that problem. If the open decision is what to ask about a collection, selected content may be more informative than another completed metadata field. The review should be designed to help formulate the request, with a clear account of what it is trying to discover.

Take a separate, invented exploratory request for the same fictional support archive. Assume a researcher wants to know whether resolution notes contain useful accounts of workarounds. The company, researcher, purpose and possible findings are all hypothetical. A date range and record count cannot reveal how those workarounds are described, or whether the notes distinguish a temporary fix from a lasting resolution.

An authorized selection of relevant records could help the researcher decide what should be profiled. If the notes distinguish the action taken from the eventual outcome, those may become useful fields to describe. If the distinction appears only in a sequence of updates, the subsequent request may need that sequence. If the selected records contain neither, the researcher has a reason to reconsider the proposed use or ask whether a different part of the archive contains the evidence. These are possible interpretations of a future inspection, not findings about an actual collection.

Exploration wins here because the uncertainty concerns the question itself. It does not need to wait for the owner to complete a detailed specification whose relevance is still being discovered. It does need a bounded purpose, a stated selection and an authorized release. Rightmark's sample guidance provides that release boundary; curiosity alone does not define the material or access required.

There is also a point at which exploration should stop being the explanation for every new request. Once the reviewer can name the useful characteristic, put it into the profile discussion and ask what evidence would establish it. If the requirement becomes a particular year or a particular relationship, assess that requirement directly. Appealing examples from a different period cannot repair the known absence in Request A.

Before preparing another export, attach a short decision note to the profile. It should connect four things: the dated buyer condition, the owner fact and its source, the finding those support now, and the next evidence that could change it. Name who can provide that evidence. This is a practical way to keep a sourcing conversation from becoming a growing pile of files with no settled question.

For Request C, the note could read as follows. It continues the invented example and records assumptions and proposed work, not measured results:

Condition: The buyer requires tickets linked to recorded resolutions. Confirm which relationship the buyer needs to inspect and record the date of that confirmation.

Current evidence: The owner reports shared ticket IDs. Link coverage and the meaning of the associated records remain unverified.

Finding: Fit remains open on this condition.

Next action: The owner's technical lead checks the links within the agreed scope and explains unmatched or multiple matches. The buyer then identifies any remaining content question for an authorized sample review.

That note gives each side a different job. The buyer supplies the meaning of the requirement. The owner establishes what its records contain. Neither has to pretend the other question is answered to keep the discussion moving. If the owner cannot perform the proposed check, that limitation becomes the next issue to resolve, rather than a fabricated zero or an implied pass.

Rightmark's data card provides a place to describe the collection, evidence and known gaps; its request brief provides a place to describe what the buyer seeks. The decision note connects them. It should be revised when the requirement changes, records are found or a measurement is corrected, with the reason retained beside the new finding.

If nobody can say what evidence would change the current decision, asking for more files has no clear endpoint. Clarify whether the conversation is exploratory, whether a condition is unresolved or whether this collection simply misses the defined request. Each answer leads somewhere different.

A useful profile makes the next question smaller. It can show an owner why a collection misses one request, remains a lower-priority possibility for another or needs a particular answer before review can continue. That is enough to make the first exchange valuable, without treating a complete form as acceptance or an incomplete one as rejection.

Use Rightmark's data card to record the profile and its gaps, then add the one question the next evidence should answer.