A good-looking sample is not an acceptance test

A sample has a history before it reaches the buyer. It might come from the easiest system to export, the material already cleared for review or the part whose ownership paperwork is simplest to assemble. Those can be sensible choices. They also determine which questions the sample can answer.

In a November 11 preparation request, I asked for repositories with fewer contributors: “Choose ones with fewer people in their history”. Working on Rightmark's owner intake, I was proposing a starting point because gathering the rights paperwork would take time. That is a reason to select candidates. It says nothing about whether those repositories later passed a technical test or represented the rest of an owner's code.

The distinction matters even when everybody describes the selection honestly. A buyer may hear “pick whichever records you want to audit” and reasonably value that freedom. But freedom within the supplied pool does not tell the buyer how much of the collection sits outside it.

Consider an invented support-record collection. The collection, groups, counts, selection rules and results below are all assumptions, not a Rightmark transaction. Suppose it contains four disjoint groups of 20 records. The supplier chooses ten recent, English, single-message records for a first sample, support-s1. The buyer randomly selects five of those ten for a format check.

Assume all five audited records parse under the declared schema. That answers a useful question about those five records. It leaves the other three groups untouched. Auditing all ten supplied records would remove uncertainty about their format, while leaving exactly the same gap in coverage.

The missing step is to connect the sample's selection rule to the proposed purchase. If the buyer only wants recent English single-message material, the other groups may be irrelevant. If the buyer wants the entire collection for work that depends on conversational context and Spanish coverage, those exclusions belong in the decision.

This is why I would ask for the selection history before asking for more records. The owner can record how the sample was selected, including the source version, filtering and exclusions. That lets both sides distinguish “we have not inspected this group” from “this group is outside the offer.” The first calls for a judgment about missing evidence; the second changes what is being bought.

NIST's acceptance-sampling guidance describes random selection from a defined lot and plans that establish conditions for accepting or rejecting that lot. It is industrial inspection guidance, not an AI dataset standard. Its useful boundary here is the connection between the material sampled and the lot being decided on. A buyer-selected draw from a supplier-curated pool cannot silently acquire the scope of a draw from the larger delivery.

Once the selection boundary is clear, the next question is what passing actually means. A file can be easy to ingest and still leave the buyer unable to judge the task it was acquired for. The danger is a test whose answer is precise while its purchasing implication remains vague.

Stay with the invented support-s1 sample. One of the five records, ticket-05, contains an outcome and a reference to thread-19. It parses correctly. The sample's exclusion note says the thread was omitted; assume that note is accurate. A reviewer trying to understand why the support case reached that outcome cannot inspect the preceding conversation.

There are two materially different readings of this result:

The second row does not make the first result false. Nor does an omitted sample thread establish that the full collection lacks it. The buyer has found a limit in the supplied evidence. Whether that is a failed requirement or an unanswered question depends on what the sample was meant to demonstrate.

That difference changes the request to the owner. “Send a higher-quality sample” gives no clear repair target. “Can we review the approved context behind this outcome, and does the proposed delivery preserve that relationship?” identifies both the missing artifact and the decision it would serve. The owner can explain an exclusion, propose a permitted alternative or say that the requested relationship is outside the available material.

Model evaluation adds another question. Google's guidance on dividing datasets says a test set should represent both the overall dataset and the real-world data the model will encounter. That is guidance for evaluating models. It does not turn a schema pass into a commercial acceptance criterion, or establish that a purchased collection will improve a particular system.

For our buyer, a format check could justify spending engineering time on a permitted task evaluation. A task evaluation might then examine whether the supplied conversations contain the context needed to judge outcomes. Those are successive questions, with different evidence. Calling both “sample approval” loses the reason the second is needed.

Permission also affects which test is available. Describing a planned model experiment in a request does not itself authorize running it. Rightmark's buyer request brief makes intended use part of scoping and leaves allowed use to the applicable review and final license. The buyer should ask for the evaluation it needs explicitly, rather than treat access to a sample as permission for every experiment.

Suppose the owner approves adding thread-19 to a revised sample, support-s2. In this invented continuation, the buyer checks the link from ticket-05 and can now retrieve the thread. The repair has answered the immediate complaint. It deserves to be recorded as a successful correction of that relationship in that version.

The important question is where the correction happened. Did someone add the missing thread to the small sample file? Did the export rule change? Was the full delivery regenerated using that rule? A screenshot showing the repaired example cannot answer all three.

This is a propagation problem. The original defect may have come from deliberate sample exclusions, from an export rule that omitted related records, or from an isolated assembly mistake. The same visible fix can sit on top of any of those causes. Its significance depends on which cause was addressed and which output was checked afterward.

These are alternative branches, not a claim that every action took place. In particular, the five successful format checks belong to support-s1. The new observation on support-s2 is the single relationship check. Carrying the old results forward automatically would hide what was actually rerun.

The third branch is also a useful stopping point. If the agreed requirement is that every required reference resolves, a correct check of every required reference in the identified delivery can settle that requirement. Another random sample would add nothing to that specific conclusion. Whether the linked conversation contains useful context remains a different question.

There is a related but distinct lesson in model debugging. Writing about AdaTest in 2022, Scott Lundberg and Marco Tulio Ribeiro, then Senior Researchers at Microsoft Research, explained what happens when tests are used to fine-tune a model: “they no longer represent test data but rather training data”. Their account of the debugging loop describes retesting with new, adapted tests to find problems left behind or introduced by a fix. The known tests remain useful for correcting the discovered behavior; their role has changed.

Fixing a dataset export is not model fine-tuning. The practical connection is narrower: repeat success on a case used to make a correction tells you that the correction works there. To draw a broader conclusion, identify the broader object and the check that supports it. For support-d2, that could be an exhaustive structural check. For a claim about varied conversation content, it requires evidence that actually reaches that content.

An owner should be able to close a resolved issue instead of resending the same repaired example indefinitely. A buyer should be able to see whether the correction reaches the material it will receive. Naming the version and the condition gives both sides a place to stop arguing about the word “fixed.”

The strongest objection to this argument is commercial: a buyer does not need to eliminate uncertainty before making a good purchase. It may already know the supplier, have a useful view of the material and face little downside if part of the collection proves unsuitable. More evaluation can cost more than the problem it is supposed to prevent.

That objection wins when the uncertainty is genuinely affordable. Consider a separate invented purchase. Every amount and condition here is an assumption, not a market price, Rightmark offer or actual deal. The buyer is deciding whether to license the whole identified lot after seeing a curated sample.

Under these assumptions, buying the whole lot can be the better decision. Spending another $1,500 solely to avoid a loss capped at $1,200 would cost more than the loss it could prevent. The buyer is accepting the remaining quality risk. It does not need to recast five passing sample records as proof about the entire collection to justify doing so.

The calculation changes if the apparent cap leaves something out. A missed launch, an engineering team diverted for weeks or a commitment to a downstream customer can make a cheap license an expensive dependency. Equally, an evaluation may produce knowledge useful for many future purchases. Its value would then extend beyond avoiding this one loss. Those belong in the decision before comparing the two totals.

A curated sample can also be exactly the right tool for a narrow question. In their 2020 CheckList work, Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin and Sameer Singh organized behavioral testing around capabilities and test types. The accompanying research repository describes deliberately constructed tests for particular model behaviors. Their usefulness comes from targeting a question; they are not presented here as a random estimate of a supplier's data quality. Directed checks can uncover a specific problem that a general accuracy result leaves hidden.

For an export question, a manifest or relationship report may likewise be more useful than a random pile of raw records. Rightmark's metadata-only export sampling guide starts with the least revealing artifact that can answer the named question. If the question is whether required identifiers survive export, ask for evidence about those identifiers. More sensitive content is not automatically better evidence.

Nor does this argument rule out accepting a defined lot through sampling. NIST's guidance explicitly describes acceptance and rejection plans for that purpose. The distinction is a plan whose scope and decision conditions fit the lot, versus a curated demonstration being asked to carry an undefined conclusion about a larger delivery.

My objection is to that undefined conclusion. A buyer may have enough evidence for a narrow requirement, or sound reasons to buy with uncertainty. In each case, the decision has a reason beyond “the sample looked good.”

A useful evaluation record should make the next request easier to explain. It needs the material checked, the question asked, the observation and the decision that follows. If the next action is simply “send more,” it is worth checking whether the buyer has identified what more would resolve.

Return to the invented sample-only repair, where support-s2 fixes one link. The alternative fully checked support-d2 delivery has not occurred in this branch. Here is the decision note I would use:

The proposed artifact has a job. Ten more easy single-message records would not do it. If an approved relationship report answers the open question, begin there. If understanding the outcome genuinely requires conversation content, explain why that content is needed and agree the allowed scope before requesting it.

Rightmark's sample-pack checklist calls for owner approval of the exact sample, recipient, purpose and transfer path. That creates a useful discipline for both parties: the buyer states the decision it needs to make, and the owner can respond to a bounded request instead of guessing what will satisfy an open-ended quality review.

The note also prevents an evaluation from becoming an endless hunt for certainty. Once the relevant condition is established for the agreed material, close that question. When it remains unresolved, choose deliberately between obtaining evidence, accepting the uncertainty or declining the purchase. Each is more useful than preserving a vague “approved sample” label while the team disagrees about what it approved.

A clean result should move a decision forward by exactly the distance the evidence supports. Sometimes that is another evaluation. Sometimes an agreed requirement has been met, or the buyer has chosen a purchase with a known limit on the downside. The owner and buyer should be able to name which one before the small, polished sample becomes a commitment to the larger collection.

Use the buyer request brief to describe the data you need and the question your next evaluation must answer.