Flaky Test Quarantine: Owners, Expiry, and Nonblocking CI Results
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A checkout defect report says that a cleared apartment value reappeared on the final order. Repeating the original steps may confirm a fix, but it leaves another question unanswered: could a different path through address editing preserve the same stale value?
Error guessing uses experience with past behavior, common mistakes and similar systems to anticipate possible failures and design tests that expose them. ISTQB places it among experience-based testing techniques. Its fault-attack approach starts with a list of plausible errors, defects or failures and turns that list into tests.
For a manual tester, the useful output is a specific failure hypothesis, a reproducible probe and a record of what was observed. Historical evidence supplies the reason to investigate. Current requirements supply the expected outcome. Neither a familiar symptom nor a plausible explanation proves that the current build contains a defect.
Start with a history window that another tester can identify. Record the product area, releases considered, report references and evidence available. A checkout-focused review might include resolved checkout defects and related support reports from the releases affected by the current change. Exclude unrelated incidents rather than building a catalog that nobody can finish reviewing.
The following packet is entirely fictional. It contains only checkout reports H-A, H-B and H-C from fictional releases R-A and R-B. The reports, requirements and probes below are illustrative, not customer evidence. No checkout tests have been executed, and every current result is Not run.
Historical Report | Observed in the Fictional History | Pattern to Investigate |
|---|---|---|
H-A, release R-A | Cleared apartment value reappeared on the order | Stale address state |
H-B, release R-B | Repeated submission produced duplicate orders | Repeated-action handling |
H-C, release R-A | Duplicate report of the H-A incident | Merge with H-A, retain its evidence |
Keep symptoms separate from explanations. H-A describes a value that reappeared; it does not establish whether the cause was a stale browser model, an omitted update or a server-side merge. Those are alternative explanations to investigate, not confirmed diagnoses. If a real report contains a verified cause, retain its supporting analysis and the version to which it applies.
H-C should remain linked to H-A so its evidence is not lost, but it should not count as an independent occurrence. Likewise, multiple reports against a frequently used feature do not by themselves establish a higher defect probability. Usage, reporting practices and earlier testing all affect what enters the history.
In a real review, use approved, sanitized report extracts. Remove customer identifiers, payment data, credentials and unnecessary attachments. Preserve enough context to understand the trigger, affected state, expected behavior and observed failure. Mark missing details as unknown instead of completing the story from memory.
A useful hypothesis connects a historical signal to a condition that could still expose a failure. For H-A: when a previously populated apartment field is cleared during checkout, an older value may survive in the stored order address. This states the condition, the suspected failure and the observation needed to investigate it.
“Try unusual addresses” is too broad to guide execution. “The address cache is broken” assumes a cause that the packet does not establish. A failure hypothesis should be specific enough to challenge without forcing the tester to confirm a preferred explanation.
For this fictional store, the reviewed requirement set is:
These are declared example requirements, not rules that every store necessarily implements. A real team must resolve its own address persistence and repeated-submission behavior before using the outcomes as test oracles. An oracle is the basis for deciding whether the observed behavior is acceptable.
Keep the source of a test idea separate from its oracle. Past behavior can suggest a stale-state probe, but an old defect cannot define correct behavior. If requirements conflict, record the ambiguity and ask the responsible product owner to resolve it. A tester should not silently choose whichever outcome the current application produces.
Compare each proposed probe with existing tests. A requirement-derived case may already cover clearing the field. Reuse or extend it when the starting state and observations match; add a separate probe when back navigation or delayed confirmation introduces a genuinely different condition. Keep equivalence partitions, boundary values and state-transition coverage in the test design. An experience-based list does not demonstrate that those obligations have been met.
Use an isolated, approved test environment with synthetic accounts and orders. For this design, payment and fulfillment must be disabled or replaced with approved test substitutes. The tester also needs an authorized read-only way to inspect stored order addresses and order identities. A success message alone cannot establish those outcomes.
Record the selected build, browser, environment configuration and requirement version before execution. Reset the account and cart between probes, and retain the starting state for each attempt. If the environment cannot expose the required observation safely, mark the probe Blocked when execution is attempted; do not infer a pass from the visible screen.
P-A follows H-A and its duplicate H-C. The hypothesis is that clearing an optional field may leave a previously saved value in the order address. Prepare a synthetic account whose saved address contains Apartment 7, with a fresh cart and no existing order for this checkout attempt.
The expected outcome under CHK-1 v1 is an empty apartment value at review and on the stored order. Other address fields should remain as submitted. If review is correct but the stored order retains Apartment 7, record that contrast; it narrows the observed failure without proving which component caused it.
Current result: Not run. The setup and expected outcome are planned, not observations from an application.
P-B tests another path suggested by the stale-state pattern. Its hypothesis is that returning from review to address editing may restore an older address snapshot. Start with a fresh synthetic checkout whose saved street is Test Street A.
Under CHK-2 v1, Test Street B should appear at the final review and remain on the stored order. This probe differs from P-A because the suspected trigger is navigation and resubmission rather than clearing an optional value. Keep the other inputs stable so a failure remains interpretable.
Current result: Not run. A future pass would apply to the recorded path and environment, not every possible navigation sequence.
P-C follows H-B. The hypothesis is that repeated final submission while confirmation is delayed may create another order for the same logical checkout attempt. The test requires an approved isolated fixture that delays confirmation while allowing the initial submission to be processed. That fixture is a prerequisite, not something assumed to exist.
CHK-3 v1 expects only one order for the logical attempt. If the interface prevents another submission, record that behavior instead of inventing a second accepted action. If confirmation never resolves or order identity cannot be inspected, the expected outcome cannot be established.
Manual repetition does not prove simultaneous processing, server-side idempotency or behavior across devices. Those may require separately designed tests. This probe addresses the observed manual interaction under the recorded delay conditions. Current result: Not run.
A test design can contain complete expected outcomes while containing no execution evidence. Keep that distinction visible. The following is a filled planning record for P-C, not an executed result:
When a tester runs the probe, create an attempt record rather than replacing the planned expectation with a result label. Capture the actual build and environment, browser version, tester, timestamp, inputs, action sequence, expected outcome, observed behavior and evidence references. Note deviations from the plan, including a disabled button or a delay that behaved differently than expected.
A pass requires the stated outcome to be observed. Use Fail for an observed mismatch against the agreed rule and Blocked when a prerequisite prevents a valid judgment. Follow the team's agreed status definitions, while keeping a probe that has never been attempted distinct from an attempted but blocked test.
Retain each retry as a separate attempt. A later pass does not erase an earlier duplicate order or unexplained stale address. Evidence should be identifiable and accessible to authorized reviewers without exposing customer or sensitive data.
Prioritize by the consequence of the suspected failure, relevance to the current change, difference from existing coverage and ability to observe the outcome. If final-submission handling changed, P-C may deserve attention before another address variant. If the fixture is unavailable, record the blocker and escalate the untested risk rather than treating a convenient substitute as equivalent.
Agree the scope and timebox before execution. Stop when the planned investigation is complete or its agreed limit is reached. Stop sooner if test data isolation fails, an external side effect becomes possible, or the required observation is unavailable. Record unfinished probes and who must decide whether to extend testing or accept the remaining risk. Running out of time is not evidence that the risk disappeared.
For a new mismatch, retain the attempt and raise an anomaly with reproduction steps, expected versus observed behavior, build, environment and supporting evidence. Triage may identify a product defect, a requirement issue, a test-data problem or an environment failure. Update the historical pattern only when that disposition is supported; do not turn every unexpected result into another confirmed defect.
After a fix, confirmation testing checks the original failure condition. Regression selection should also consider related behavior that the change could affect. Promote a useful probe into maintained regression coverage when its prerequisites and oracle are stable, preserving its requirement reference and execution history. Keep exploratory follow-up separate when the investigation is still changing.
The remaining risk includes failures outside the selected history, untested paths, unsupported environments and unknown causes. A passed probe weakens a particular failure hypothesis under particular conditions; it does not prove the application is defect-free. Begin with a bounded historical pattern, write an observable rule beside the hypothesis, and require an honest attempt record before expanding the test list.
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A green automated test history can help a developer identify a safe-looking change while leaving a QA lead unable to answer a different question:…
A sprint can contain carefully estimated testing work and still overload the only person who can perform it. The missing input may be a…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.