Flaky Test Quarantine: Owners, Expiry, and Nonblocking CI Results
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A feature can be marked “checked” even when the tester has examined only its most familiar path. The next tester may interpret the same label differently, choose different data and leave a different set of behaviors unexplored. A reusable testing checklist helps only when its entries identify meaningful conditions and the execution record shows what was actually tried.
Checklist-based testing is an experience-based technique in which testers design, implement and execute tests to cover conditions listed in a checklist. The checklist guides what to examine without necessarily prescribing every input and action. Its usefulness depends on the quality of those prompts, the tester’s judgment and the evidence retained after execution.
“Test the catalogue” names an area, but it gives little guidance about the behavior that deserves attention. “Does title matching ignore letter case?” identifies a condition that can guide concrete tests. A tester can choose suitable titles, vary capitalization and compare the observed results with the matching rule.
The ISTQB Foundation Level syllabus, authored and copyrighted by ISTQB and its syllabus authors, describes checklist items as separately and directly checkable. Its checklist-based testing guidance also cautions against overly general entries and items better treated as entry or exit criteria. “The environment is ready” and “release approval obtained” may belong in process checklists, but they do not replace feature-level test conditions.
A useful prompt identifies a bounded behavior and has an identifiable basis for judging it. Avoid combining unrelated checks such as “filters work, results are fast and the layout is accessible.” A single tick cannot explain which condition was investigated or which failed. Separate the concerns, then decide which belong in the present testing scope.
A checklist may include an expected condition or a reference to one. The distinction from a scripted case is the level of execution detail, not a rule forbidding expected results. A short question still needs a test oracle: the requirement, approved example or other justified reference used to decide whether an observed result is correct. The current implementation’s behavior is not sufficient authority merely because it is what the tester sees.
Consider a fictional catalogue with these entries: Robin and Heron belong to Birds; Fern belongs to Plants. This is an unexecuted design illustration, not a runnable application or a report of testing performed. The checklist baseline is CATALOG-CL-1.0, and its test basis is CATALOG-REQ-1.0.
The fictional requirements deliberately keep the behavior narrow:
The planned scope covers category filtering, title filtering, empty feedback and the visible result count. It excludes sorting, pagination, performance, security, accessibility conformance and cross-browser compatibility. Non-ASCII matching and maximum query length remain unspecified. Those exclusions and unknowns are visible gaps, not implied approvals to ignore them in a real project.
For future execution, a tester would record the isolated build, browser and chosen data before starting. Neither a real environment nor an execution owner has been selected here. A proposed checklist maintainer would keep the baseline current, but the actual acceptance authority and review decision remain unknown. Version labels, roles and recording fields in this example are local conventions, not mandatory ISTQB fields.
The checklist below is reusable across future runs that share this basis. Category prompts use an empty title query, while title-query prompts use category All. Each entry addresses a condition rather than a fixed sequence of clicks.
Item | Condition-Level Prompt | Basis and Risk | Design Status |
|---|---|---|---|
C1 | For the selected category, are all matching fixture entries available? | R1: Missing or unrelated titles. | NOT RUN |
C2 | Does the selected category exclude entries from other categories? | R1: Missing or unrelated titles. | NOT RUN |
C3 | Does selecting All restore the category-unfiltered catalogue? | R2: A previous filter remains active. | NOT RUN |
C4 | Does title matching ignore ASCII letter case? | R3: A valid title is hidden by case differences. | NOT RUN |
C5 | Does an unmatched title query show No matching titles? | R4: An empty result is mistaken for unfinished loading. | NOT RUN |
C6 | Does the displayed count equal the visible entry count? | R5: Misleading result totals. | NOT RUN |
R1 produces separate inclusion and exclusion prompts because a filter can omit a valid entry or retain an unrelated one. These are different failures. R4 also requires an empty visible list; C5’s feedback prompt alone does not cover that entire requirement. Record that omission and add a separately checkable list-emptiness condition before claiming full R4 coverage. This baseline is intentionally a bounded starting point, not a complete requirement-coverage claim.
C4 leaves the tester room to choose a query and a title. A fully specified case fixes those choices so that someone else can repeat the same investigation. Both representations can refer to the same requirement and expected behavior, but they preserve different amounts of execution detail.
For fictional case TC-TITLE-CASE, load exactly the catalogue described above, select All, leave the title query empty and apply no other filters. The case uses CATALOG-REQ-1.0, requirement R3. Its input is the query “rObIn.” The planned steps are:
The expected visible membership is Robin only, identified as ROBIN in the fixture. Heron and Fern must not appear. This result is derived independently from R3 and the fixture titles: case-insensitive comparison makes “rObIn” match Robin, and neither other title contains that query. The order of results is not asserted because this case’s requirement does not specify sorting. Its execution status remains NOT RUN.
The case does not establish every aspect of title matching. It fixes a particular mixed-case input, category and starting state. To investigate C4 more broadly, a tester could choose additional ASCII case variations or another fixture title and record those choices. For exact reproduction of a reported failure, preserve the specific input and action sequence rather than relying on the original prompt alone.
For a future C1 investigation with Birds selected, R1 makes Robin and Heron the expected matching membership. C2 asks separately whether Fern is excluded. For C5, a query such as “zzzz” has no match in the fixed catalogue, so the feedback oracle comes from R4. Checking that feedback would not by itself verify the empty-list condition omitted from this checklist.
C6 needs its own observation: compare the displayed count with the entries actually visible after applying the chosen filter. Do not assume that an apparently correct list proves its count is correct. No application has been exercised for any of these examples; they describe future tests and their requirement-derived oracles.
Keep the reusable prompt in the baseline and record each execution separately. A useful local record includes the checklist and basis versions, item ID, build, browser, category or query, expected-condition reference, observed result, evidence reference and executor. Record questions or defects alongside the relevant observation. This detail makes it possible to understand what a short prompt became during a particular run.
A focused checklist can make useful testing knowledge reusable. Someone who remembers an earlier category-filter omission can turn that concern into a prompt that colleagues can revisit. The team gains a common reminder without documenting every possible navigation path in advance. The prompt still needs review against the current feature rather than being treated as permanently valid because it once found a defect.
Condition-level prompts also leave room for adaptation. A tester may notice an unexpected state while investigating a filter and choose a relevant follow-up test. This flexibility can broaden the investigation, particularly when experienced testers understand the product and its risks. It does not guarantee better defect detection, and the resulting choices need recording if others must interpret or reproduce them.
For a new team member, the checklist can reveal which behaviors experienced colleagues consider important. However, it cannot supply all missing domain knowledge. Review the basis together, clarify ambiguous prompts and compare sample execution notes before assuming that everyone interprets an item consistently. Any reduction in documentation or onboarding effort is conditional; this example measures neither time saved nor effectiveness.
Short prompts move some test design into execution. This creates useful freedom and a responsibility to make the resulting investigation visible. The ISTQB syllabus notes that high-level checklists can lead to variability, with potentially greater coverage but less repeatability. Choosing the level of detail is therefore a decision about the testing objective, not a contest between universally better formats.
If testers disagree about what C4 means, first check the requirement and scope. One might investigate ASCII case variations while another assumes support for every writing system. The frozen basis supports the former and leaves the latter unknown. The appropriate response is to clarify or obtain the missing requirement, not silently treat the wider interpretation as an approved acceptance rule.
When the same inputs and actions must be repeated, preserve a scripted case or detailed execution procedure. When the objective is broader investigation of a familiar condition, a checklist prompt may be sufficient if the tester records the choices and observations. Requirements for formal evidence or repeatability, including applicable regulatory or contractual requirements, still need their own assessment; a checklist label does not waive them.
A completed checklist shows what was recorded against its entries. It does not prove that every requirement, risk, input combination or user need was covered. The missing empty-list prompt for R4 makes this concrete: marking C5 complete could leave stale results unexamined even when the empty-feedback message is correct.
Compare the checklist with the test basis and risk inventory to identify omissions. Keep unexplored conditions, excluded areas and unanswered questions visible in the test report. A completion percentage cannot supply the missing coverage definition, and a pass means only that the observed result met the stated oracle under the recorded conditions. It cannot establish absence of defects.
Experience-based testing can coexist with other techniques. Use requirements analysis and appropriate structured techniques to identify additional conditions, then use exploratory testing to investigate discoveries and areas that need learning. Exploratory testing can have charters and session notes; it should not be dismissed as necessarily undocumented. Detailed cases and existing automated checks can also remain part of the testing approach, with their own evidence rather than duplicate checklist ticks.
Reusing CATALOG-CL-1.0 means reusing its conditions and assumptions, not copying its old execution status into a new run. Review the checklist when requirements, risks or relevant defect information change. The primary syllabus specifically recommends regular defect-informed updates and warns against letting checklists become too long.
Suppose a later requirement changes title matching to a different rule. Review C4, its oracle, the scripted case and any related evidence references together. Issue a new checklist baseline where necessary, retain the old version for interpreting historical runs and make the change rationale visible. Renaming a form field without examining behavior does not create coverage of the new rule.
Retire items that no longer apply, but preserve why they were retired and which baseline contained them. Add new conditions when justified, separate compound prompts and remove duplicates that add no distinct investigation. For the current example, the missing empty-list condition belongs in the next review, with its omission recorded rather than hidden behind a completed feedback check.
Before the next real run, ask whether the checklist’s basis still applies, whether each prompt has a usable oracle and whether its omissions are acceptable to the actual decision-maker. Keep prompts concise where experienced judgment is useful, and add scripted detail where reproducibility requires it. A reusable checklist earns its place by guiding a defensible investigation, not by accumulating ticks.
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A green automated test history can help a developer identify a safe-looking change while leaving a QA lead unable to answer a different question:…
A sprint can contain carefully estimated testing work and still overload the only person who can perform it. The missing input may be a…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.