A CI dashboard turns green. An agent benchmark records PASS. A release workflow completes its automated checks. The tempting conclusion is that the software is safe, the agent is correct, and the deployment has been justified.
That conclusion is larger than the result. A pass is a bounded evaluation verdict: for an identified subject, under defined conditions, a particular evaluator observed something that met its acceptance rule. Whether the result supports a broader claim depends on the oracle, test coverage, execution record, artifact identity and assumptions about the operating environment.
A valid PASS can be valuable evidence. It is not meaningless, and this note does not argue that automated testing should be distrusted. The error is to silently turn evidence for a narrow proposition into a certificate for a different, much wider one.
This is not a repetition of A Trace Is Not Evidence. That research note asks whether a recorded execution establishes an external effect. Here the problem begins even if the test ran correctly and its pass result is authentic: what exactly was tested, and how far may its conclusion be generalized?
The proposition inside a PASS
Imagine a customer-support agent evaluated on a fixture marked risk_level=high. The oracle checks that the agent's final response contains ESCALATE. The response includes the marker, and the test runner reports PASS.
Assuming the fixture, runner and assertion are trustworthy, the immediate conclusion is:
On the supplied example, the evaluated configuration produced an output that satisfied the specified escalation-marker assertion.
It does not follow that the agent independently recognized risk correctly, that a human received the escalation, or that every high-risk request will be treated properly. A real external handoff, if it matters to the claim, needs its own suitably authoritative check.
A test may pass correctly while an interpretation of its result is false. The pass refers to an assertion, not to an unrestricted property of a system. Nor does one failure demonstrate that every other behavior fails.
A legitimate pass for an insufficient test
Consider an illustrative, not experimentally observed, release candidate for a support agent. Its evaluation replays 120 stored conversations. The acceptance condition is that for fixtures bearing risk_level=high, the generated answer must include ESCALATE. All 120 tests pass.
Three limitations remain. The risk label is supplied by the fixture rather than independently derived by the model under test. The marker is a text property rather than evidence of successful routing to a real recipient. The replay set may not cover new languages, contradictory customer records, adversarial prompts, absent metadata, policy changes or live integration failures.
The test suite can be internally correct and still insufficient for a decision to deploy the system into those untested conditions. Conversely, these gaps do not prove that deployment would fail. They mean the broad claim has not been established.
This distinction is important for ZeroField's scientific identity: no real ZeroField evaluation is claimed, no system has been measured in this article, and the example does not assert a numerical defect rate.
Five boundaries behind a successful verdict
A review should keep five elements separate.
- The subject. Identify the model, executable artifact, prompts, tools, policies, configuration, dependencies and version evaluated. A mutable branch name is not the same as an immutable tested subject.
- The conditions. Document fixtures, environment, permissions, mocks, time, input distributions and failure modes included or excluded.
- The oracle. State the actual predicate for PASS: schema match, rubric score, tool-call pattern, policy decision, observed receiver state or some combination.
- The execution record. Establish that the claimed tests ran, how missing and skipped cases were treated, and that the report corresponds to the expected subject and run.
- The inference. Name the proposition a release decision is supposed to accept, plus the assumptions and residual uncertainty needed to extend the observation beyond tested conditions.
A missing link need not make the test result false; it limits what the result proves. NIST's AI Risk Management Framework MEASURE function emphasizes uncertainty, validity, reliability, independent review, and documenting limitations of generalization beyond the technology's development conditions [1, 2].
The oracle can pass while the intended behavior fails
Oracles have different jobs. A syntax oracle can establish conformance to a schema. A task oracle can test a chosen behavior on an example. A policy oracle can check a rule at a defined authority boundary. A receiver check can examine a resulting external state, with whatever freshness and consistency guarantees the receiver provides.
These checks do not answer identical questions. The word ESCALATE might satisfy a format assertion without causing an external escalation; a successful mock-tool call may leave a real integration untested. A learned evaluator may also have uncertainty, sensitivity to its scoring rubric, or errors requiring independent review.
A stronger oracle is not automatically required for every purpose. If the stated claim is merely “this JSON matches a schema,” schema validation may be sufficient. Trouble starts when the result is used to assert “the system is safe and the customer received help.”
The distinction is adjacent to the trace-versus-outcome problem, but the failure addressed here is at the definition and generalization of an acceptance criterion.
Coverage is a boundary, not a fraction of reality
A result such as “120/120 PASS” gives a denominator for the chosen tests. It does not establish that the tests represent all intended tasks or hazards. Repeating many easy cases may increase the count without covering a rare but consequential failure mode.
Meaningful coverage should be organized around the intended operational domain: languages, user categories, integrations, privilege boundaries, policy conditions, harmful actions, stale context, timeouts and unforeseen instructions. Teams can use representative samples, deliberately adversarial cases, integration tests and monitoring, but their results must still be interpreted against explicitly identified limits.
A finite test suite does not prove the absence of all defects. Formal verification can establish properties under a defined model and assumptions, but it should not be confused with ordinary test-suite success. NIST guidance calls for suitable evaluation methods and documented uncertainty; it does not provide a universal green-score threshold that certifies every AI application [1, 2].
Was the artifact that passed actually the artifact released?
Suppose a build with immutable digest A passes all planned tests, but deployment later resolves a mutable tag to digest B. The test may remain valid for A while saying little directly about B.
That question is separate from whether the release request was authorized. Human Approval Boundaries examines the authority relationship. Here the concern is whether a test verdict is bound to the artifact and configuration named by the release claim.
SLSA provenance describes where, when and how software artifacts were produced. Its verification guidance asks consumers to compare artifact identity, builder, source and declared parameters with trusted expectations [3, 4]. That can strengthen evidence of origin and build identity. It does not prove that the program works correctly or is safe for a new environment.
GitHub status checks likewise concern particular commits and workflows. GitHub documents conclusions including success, skipped and neutral; the meaning of green depends on which jobs actually ran and which rules treated a result as acceptable [5, 6]. A release gate must not silently equate “mergeable” with “all relevant risks absent.”
A matrix for interpreting green results
The same visible PASS can support different propositions. The claims below are conditional on the run and its inputs being genuine; none are certification statements.
| Observed PASS | Narrow claim supported | Not established by this result alone | Additional evidence needed |
|---|---|---|---|
| Schema validation | Selected outputs match a defined structure | Meaning, policy compliance or external effects | Domain-specific task and policy checks |
| Replay suite | Chosen fixtures satisfy chosen oracles | Reliability across omitted conditions | Coverage rationale, edge and adverse cases |
| Model-based grader | The configured evaluator assigns a passing result | Human-equivalent or error-free judgment | Calibration, disagreement review, uncertainty |
| Mock integration | A test double satisfies its contract in the fixture | Actual receiver behavior | Integration contract and receiver evidence |
| CI checks | Included jobs reached allowed conclusions | Every conceivable test ran or every hazard was excluded | Required-check inventory, skip review, artifact binding |
| Verified build provenance | Recorded artifact/build identity meets expectations | Functionality, safety or authorization | Behavioral tests, policy and risk decision |
| Canary release assessment | Observed limited rollout meets configured thresholds | Absence of failures outside measured signals | Monitoring, representative exposure, rollback plan |
| Human approval check | A scoped approval rule is satisfied | Unbounded permission or completed external outcome | Effective-resource authorization and outcome checks |
The distinction between these rows is not a ranking of good and bad tests. The question is whether the evidence type matches the specific proposition for which it is offered.
Skipped, stale, missing and inconclusive are not synonyms for PASS
A dashboard may summarize several different states in one color. Some jobs can be skipped by condition while downstream or required-check semantics still treat the result as acceptable. GitHub documents such cases and also explains that required checks apply to the relevant, sufficiently recent commit [5, 6]. A green interface is not by itself an inventory of executed assertions.
A rigorous reviewer distinguishes executed-and-passed, skipped by a stated rule, not run, inconclusive, timed out, and failed. That does not mean skipping a test is always wrong. An irrelevant test can be explicitly waived by policy. What matters is that the exception is understood and the release claim is no broader than the evidence left after the exception.
Freshness matters too. A pass from yesterday may remain relevant if no material premise has changed, but a new artifact, policy, integration or input population can invalidate a needed inference. Re-running every test for every trivial metadata change is not necessarily justified. The evaluation contract should identify changes that affect the acceptance proposition.
What a defensible release gate would record
A practical claim-bound gate need not require a complex centralized platform. It does require traceable answers to ordinary questions:
- Subject: Which immutable artifact, model, prompt, tools and configuration were evaluated?
- Intended use: Which tasks, users, environments and consequential actions are being considered?
- Claim: What behavior or risk property is being asserted, and at what level of confidence?
- Oracle: What observable result actually qualifies as a pass, and what does it ignore?
- Coverage: Which representative, adverse, edge and integration conditions were exercised or omitted?
- Status semantics: How were skips, missing checks, failed tests and indeterminate outcomes represented?
- Authority: Who may decide that remaining gaps are acceptable for this particular release?
- Continuity: Does the deployed system match the evaluated subject, and what monitoring and recovery are in place?
The Google SRE Workbook describes canarying as a limited deployment evaluated over a chosen exposure and time to support a rollout decision [7]. It adds a different form of evidence beyond pre-release tests, but a canary pass also remains bounded by traffic, duration and monitored metrics. NIST's Secure Software Development Framework describes testing and verification as part of a broader process for managing software risks rather than a single universal proof event [8].
Ten attempts to falsify an overbroad PASS claim
The following adversarial questions test the inference drawn from PASS, not whether testing has value.
- Wrong subject: Does the report identify a different artifact or configuration from the proposed release?
- Weak oracle: Can a cosmetic output pass while the claimed behavior is absent?
- Missing category: Is a foreseeable, consequential class of input or action excluded?
- Changed policy: Would a materially different policy version alter the verdict?
- Skipped check: Does the dashboard remain green though a required assertion never executed?
- Stale fixture: Do the test cases no longer represent the target operating conditions?
- Mock boundary: Is a simulated tool result misdescribed as evidence of actual receiver behavior?
- Grader uncertainty: Would reasonable rubric, seed or sampling changes overturn the verdict?
- Release substitution: Can a different artifact be deployed after the pass without invalidating the gate?
- Drift after release: Can risk change without monitoring that would detect and reassess it?
A useful finding says, for example, “the claim of successful delivery exceeds the text-only escalation oracle” rather than vaguely calling the entire test “fake.” Likewise, an unproven proposition should not be labeled a demonstrated failure. The aim is precise, falsifiable reasoning about evidence and decisions.
Where ZeroField fits — and where it does not claim results
The ZeroField research domains separate concerns often collapsed into one green badge. Framework asks what a claim means. Architecture identifies the subject, oracle, evidence and decision boundaries. Systems asks whether a particular executable and its actual interfaces satisfy those relationships. Research studies where the inference fails and which limitations remain.
Those are architectural distinctions, not descriptions of a deployed ZeroField evaluation product. This article does not assert that ZeroField operates a validated grader, a proven safety monitor, a certified build-provenance system, or a measured release gate. It presents no original experiment or performance number.
The earlier notes address adjacent but separate questions: Authority Is Not Capability asks who may perform an action; A Trace Is Not Evidence asks what recorded events establish; Unknown Is Not Failed examines uncertain external outcomes; and Human Approval Boundaries examines the scope of a human decision.
This note focuses on a fifth proposition: a successful evaluation verdict should not be allowed to claim more than the actual evaluated subject, oracle, conditions and evidence support. A precisely scoped pass is useful. A pass promoted into unearned proof is a category error.
References and technical context
- NIST, AI Risk Management Framework Core — MEASURE. https://airc.nist.gov/airmf-resources/airmf/5-sec-core/ — Validity, reliability, uncertainty, independent review and limits on generalization. Voluntary guidance, not automatic certification.
- NIST, AI RMF Playbook — MEASURE. https://airc.nist.gov/airmf-resources/playbook/measure/ — Contextual evaluation practices and documenting limitations, not a universal pass score.
- SLSA Framework, Build Provenance. https://slsa.dev/spec/v1.0/provenance — Artifact production identity and provenance semantics; not application-behavior validation.
- SLSA Framework, Verifying Artifacts. https://github.com/slsa-framework/slsa/blob/main/spec/verifying-artifacts.md — Comparing provenance and artifact binding with trusted expectations.
- GitHub Docs, Status checks. https://docs.github.com/en/pull-requests/reference/status-checks — Commit-specific checks, successful conclusions and skipped-job semantics.
- GitHub Docs, Troubleshooting required status checks. https://docs.github.com/en/pull-requests/how-tos/merge-and-close-pull-requests/troubleshooting-required-status-checks — Required-check freshness and job/workflow skip nuances.
- Google SRE Workbook, Canarying Releases. https://sre.google/workbook/canarying-releases/ — Limited deployment and measured rollout decisions.
- NIST SP 800-218, Secure Software Development Framework Version 1.1. https://csrc.nist.gov/pubs/sp/800/218/final — Testing and verification as part of wider secure-development practice.
Research scope: This note synthesizes cited guidance and uses illustrative thought experiments. It reports no new empirical study, does not certify any AI system, and does not imply endorsement of ZeroField by any cited institution.