An AI agent turns off a daily digest, clicks Save, and reports success. The browser displays a green confirmation. A reviewer still needs to know whether the setting was saved for the intended account, whether unrelated preferences changed, and whether the next session will read the saved value.
An AI test oracle is the basis for deciding whether the agent's observed result satisfies an approved requirement. It combines expected behavior, acceptable evidence, and a decision rule. The agent may choose how to navigate the browser, but completing its actions does not establish that the application reached the required state.
The test oracle problem is deciding what counts as a correct result and how to recognize it. Browser agents make this especially visible because a plausible sequence of actions can end without the intended outcome. The agent's explanation is another output to assess, not independent proof of success.
Anthropic's guidance on agent evaluations distinguishes the transcript, which records a trial's interactions, from the outcome, which is the final state in the environment. Its browser-testing discussion distinguishes a confirmation page from backend evidence that an action took effect. For a settings journey, the corresponding question is whether the right saved record contains the requested value.
Keep four responsibilities separate: the agent performs actions, an observer collects evidence, assertions compare that evidence with approved expectations, and a result policy decides the disposition. Independence means the acting agent cannot silently change the expected value, suppress a required observation, or approve its own test rewrite. A second model reading the same success narrative would not automatically provide independent evidence.
An oracle contract should identify:
These details constrain the claim. An assertion about the current input field establishes a UI property. An assertion about a committed settings record establishes a different property. Neither should inherit a stronger meaning because the runner displays a passing badge.
Consider a fictional application on build S1. Requirement SET-7 v1 says that user U-demo in tenant T-demo can turn the daily digest off without changing the locale. The controlled starting state is daily digest on and locale en-GB, with no other writer modifying that user's settings. The input is fixed: change only the digest to off and activate Save.
For this illustration, SET-7 v1 requires the change to be committed and displayed in a fresh session within 10 seconds of Save activation. That cutoff is an authored example requirement, not a measured response time or a framework default. The expected poststate is daily digest off and locale en-GB for the same tenant and user.
The example assumes a documented durable settings store S and a read-only observer O that can retrieve its committed state without relying on browser cache. O's subject identity and read-after-write semantics must be established before any real test. A separate clean-session UI check must display off for the same subject. The agent's route through menus is not part of the acceptance predicate, provided it stays within its authorized actions.
This is an unexecuted design example. No browser journey, application API, agent, or model judge was run. The following rows are authored expected cases under the fixed policy, not recorded test results.
Expected Case | Evidence Available Under the Fixed Policy | Decision |
|---|---|---|
P: Complete proof | Authoritative committed value is off, locale remains en-GB, and a fresh session displays off within the cutoff. | Pass |
F: Observed violation | Valid authoritative read at the cutoff still shows on, or the protected locale changed. | Fail |
I: Missing proof | Only clicks or a toast are available, or identity, access, timing, telemetry, or a required assertion is missing. | Inconclusive |
Case F requires a valid observation: the correct subject, trustworthy source, applicable timing, and controlled preconditions. A request that could not authenticate to the observer does not establish that the digest remained on. It establishes an evidence gap. Conversely, a valid read that still shows on at the required cutoff establishes a violation even if the agent reported success.
The UI check is a separate obligation. If the store correctly contains off but a valid fresh-session observation still displays on at the cutoff, the journey fails its display requirement. If the display cannot be observed and no other required check establishes a violation, the overall result is Inconclusive. Keep individual check outcomes so an incomplete observation does not erase an already established failure.
Exact requirements should normally use exact checks. The digest must equal off, the locale must remain en-GB, and both must belong to T-demo/U-demo. A model does not improve those comparisons by describing the screen more persuasively.
Playwright's documentation illustrates useful building blocks. Its web assertions re-fetch and retry an element until the expected condition is met or the assertion times out. An input-value assertion checks an input value; a text assertion checks text. Retrying either makes its observation more tolerant of asynchronous rendering, but does not convert that UI property into proof of durable storage.
A Save click confirms an interaction completed under the runner's conditions. A toast confirms that the UI presented a message. An HTTP success status confirms a response classification. Even a response body containing off may be an echo of submitted input rather than a committed record. The oracle must specify what the response represents before accepting it as outcome evidence.
Playwright also documents checking server-side postconditions after actions in the browser, using API requests and assertions on returned fields. That is a mechanism for obtaining another observation layer, not a guarantee about every application API. A team's chosen read interface must establish the relevant committed state, subject, and consistency behavior. A test endpoint that returns a cached projection or another user's record can produce a precise comparison against the wrong evidence.
Reloading the page can reveal a lost update, but a retained value after reload is not necessarily proof of server persistence. The browser may restore local state, reuse cached data, or read a service worker response. Starting a clean session removes some client carryover, yet the server may still serve a cache or replica whose relationship to the durable store is unclear.
For the fictional policy, the storage claim therefore depends on O's documented authority as well as the fresh-session display. Before implementation, confirm what O reads, how it identifies the subject, and how its consistency guarantees fit the cutoff. If only browser-visible evidence is available, narrow the claim to what that interface establishes or leave the durable-state obligation Inconclusive. Do not quietly replace the original requirement with an easier UI check.
The observation window also limits the conclusion. Showing that state is committed and visible by the cutoff does not demonstrate survival of every future restart, regional outage, or concurrent edit. Those risks require additional requirements and tests. Keep this journey bounded rather than expanding one passing setting change into a general reliability claim.
Keep the approved requirement, observer configuration, and acceptance comparator outside the agent's edit scope. The agent may propose a locator repair, but changing an expected value or removing a persistence check requires separate review. Preserve the original attempt and the proposed diff so a later passing run cannot conceal what changed.
This boundary is also useful during test maintenance: a repaired script must still detect the same unacceptable outcome. The related discussion of safe self-healing test automation explains why restoring execution and preserving a requirement need separate checks.
A model judge can help where correctness cannot be expressed adequately as a simple equality. In the same settings journey, a separate qualitative check might ask whether the confirmation clearly tells the user that the daily digest was turned off. That question concerns communication, not whether the durable record changed.
Give the judge a fixed rubric, the approved requirement, and the relevant captured evidence. Require a dimension-specific decision with evidence references and an option to return Unknown when the material is insufficient. Do not let the agent's own account of success replace the underlying capture. Keep the qualitative grade separate from the deterministic storage and display checks.
A fluent judge explanation cannot override a valid authoritative read showing on. Nor can a judge infer an inaccessible backend record from a screenshot. If the communication criterion becomes mandatory, define its acceptance rule before execution and include it among the required checks. If it remains advisory, report it separately without changing the journey's pass criteria.
Anthropic recommends calibrating model graders with human experts and allowing an Unknown response when information is missing. OpenAI's evaluation guidance likewise recommends task-specific evaluation, human calibration, clear rubrics, and attention to biases such as favoring longer responses or particular response positions.
For this journey, build a labeled reference set containing clear confirmations, misleading confirmations, and incomplete captures. Have reviewers apply the rubric independently of the acting agent and reconcile disagreements against the requirement. Use separate held-out examples to check whether the judge's decisions generalize beyond the examples used to refine its prompt. Inspect false acceptance and abstention cases, rather than treating aggregate agreement as sufficient evidence of safety.
Set acceptance criteria through the accountable team's risk review and retained evaluation results. No universal confidence threshold makes an arbitrary judge authoritative. A scalar confidence value is not a substitute for validated task-specific behavior, and calibration cannot recover evidence the judge never received. Revalidate when the model, rubric, requirement, or capture format changes. Until the necessary validation exists, keep mandatory qualitative decisions under human review or record them as Inconclusive.
Use a result rule that preserves the difference between an observed violation and an inability to decide. If trustworthy evidence establishes a required violation, record Fail. If every required obligation has valid evidence and is satisfied, record Pass. Otherwise, record Inconclusive and name the missing proof. An absent assertion is missing coverage, never a successful assertion.
Keep the result separate from the suspected cause. An application defect, an agent that chose the wrong control, and an unavailable evidence collector need different follow-up. A runner exception may require investigation without proving that the product violated its requirement. Conversely, a real observed product failure should not be downgraded to Inconclusive merely because an unrelated diagnostic artifact is missing.
For each attempt, retain an evidence packet with:
Store captures and raw responses under an approved access and retention policy, with secrets and personal data appropriately redacted. Hashes help bind a review to particular artifact bytes; they do not prove those artifacts describe the correct subject or a truthful state. Preserve retries as separate attempts instead of replacing missing or failing evidence with the best-looking later run.
Before trusting an autonomous pass decision, freeze the oracle contract for one journey and validate it in an authorized disposable environment against a correct outcome, a known violation, and missing evidence. The useful pass is the one whose approved conditions are supported by inspectable observations. When those observations are unavailable, keep the result open as Inconclusive and assign the next evidence-gathering action.
Two reports can describe the same symptom while disagreeing about where the failure occurs. If a reviewer accepts a duplicate suggestion because the titles…
A checkout test times out while waiting for an order confirmation. The runner reports an assertion failure, the application records a failed request, and…
A recipient-preview rule changes, but the regression suite still checks the earlier behavior. An AI agent may choose a different route through the editor…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.
