Flaky Test Quarantine: Owners, Expiry, and Nonblocking CI Results
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A release date is on the calendar, but the refund service is still changing. One tester expects to check permissions; another assumes the payment sandbox will be ready. Product expects a release recommendation, while engineering expects a list of bugs. Without an agreed boundary, each person can finish their work and still leave the release unassessed.
A software test plan records the decisions needed to coordinate a defined testing effort: what to test, how to obtain useful evidence, who owns the work, which dependencies control the schedule, and what completion means. For a small release, keep that record compact and reference detailed cases, requirements, and results rather than copying them into it.
Before drafting sections, gather the release changes, acceptance criteria, known risks, proposed delivery window, and availability of people and environments. Ask the product owner what decision the evidence must support. A plan for a permission change needs a different boundary from a platform migration, even when both have the same proposed release date.
Give the plan an identity and revision. Record the product and release, requirement revision, candidate-build reference, environment, author, reviewers, and next review point. If the candidate is not available, name its delivery owner and expected readiness point instead of filling in an invented build identifier.
The worked example below is entirely illustrative. Checkout 4.3 changes refund permissions and timeout retries. Plan CP-43 revision A proposes a six-working-day testing window. All people, hours, commitments, and later outcomes are fictional planning assumptions, not customer results or work performed in a Jira tenant.
The objective is to assess whether the changed refund behavior respects the agreed permission and duplicate-refund rules, and whether selected adjacent flows still work. Detailed cases hold inputs and expected results; the plan explains why those cases belong, who will execute them, and when their evidence is needed. Add specialist reviews or formal documentation where your organization's obligations require them. This lightweight example is not a compliance template.
Replace broad labels such as “test refunds” with observable behavior. Discuss the changed code and acceptance criteria with engineering and product, then identify affected existing behavior. Prioritize failures by consequence: an unauthorized refund or duplicate payment deserves earlier investigation than a cosmetic message change.
For CP-43, the team proposes this boundary:
Area | Planned Coverage | Boundary and Rationale |
|---|---|---|
Refund permissions | Authorized and unauthorized roles, including retry attempts | Exercise the changed permission rule; product must resolve ambiguous role expectations before execution |
Timeout retries | Retry after an uncertain response and check the resulting refund state | Investigate duplicate-refund risk using controlled sandbox outcomes |
Adjacent regression | Existing full and partial refunds, plus order-state updates | Select cases affected by the changed flow rather than promise the entire payment regression suite |
Specialist campaigns | No new load campaign, penetration test, or provider certification in this plan | Arun must confirm the separate owners and whether their evidence is a release prerequisite; exclusion here is not risk acceptance |
Translate each included row into identifiable cases or exploratory charters and expected evidence. Use scripted checks for agreed permission combinations, targeted integration checks for retry outcomes, and exploratory sessions for unexpected state transitions. Record the selected case revisions and the supported browser or service configurations. “Current browsers” is too vague for a repeatable run; the working plan needs the actual agreed matrix.
Specify the environment and data as prerequisites. This example needs an identifiable deployed candidate, refund-capable orders, authorized and unauthorized users, controlled timeout behavior, and a demonstrated data reset. Keep credentials and customer data out of the plan. Reference approved access and safe fixture records instead.
An exclusion needs a reason, an owner, and any dependency on separate evidence. If a required specialist assessment is missing, the release remains exposed even though that work sits outside this team's effort estimate. Do not relabel a selected but blocked test as out of scope to make completion look better.
Write names beside outputs, not only departments beside phases. In this example, Maya is the QA lead and one of the testers. She owns the plan, selected scope, progress assessment, and completion summary. Leo, the second tester, owns the assigned execution records and supports confirmation testing. They divide the work explicitly rather than both assuming the other will investigate a failure.
Arun owns candidate delivery, engineering triage, and the readiness forecast for corrected builds. Nia owns sandbox availability, access, and fixture restoration. Priya clarifies acceptance criteria and holds the fictional team's release authority. Maya assesses test evidence; Priya makes the release decision under that team's process. Those responsibilities can belong to different people in another organization, but neither should remain implicit.
Agree on handoffs. Arun supplies candidate identity and change notes before execution; Nia supplies a readiness record; testers attach reproducible evidence to failures. Maya escalates a missed prerequisite to its owner when it threatens the next milestone. Name a deputy or escalation route for an unavailable owner, and confirm that the deputy has the necessary access and authority.
For a team using Jira, Sherlock Test Management's current product description offers a relevant way to connect scope with execution: it describes proposed release scope using Jira Fix Version context, reviewed priorities and assignees, and human-approved versioned test baselines. The team should still challenge exclusions and assignments against the plan. These are described app capabilities, not capabilities supplied by native Jira alone. Recording Pass, Fail, or Blocked in Sherlock Runner or supported Jira panels does not establish automated browser execution, labor forecasting, or release approval.
Estimate the work required to produce evidence before assigning a finish date. Include design, readiness checks, investigation, confirmation, coordination, and reporting. Otherwise, a schedule based only on the first execution pass leaves predictable work without capacity.
CP-43 uses the following fictional QA effort budget. Developer implementation and infrastructure repair labor are excluded and owned separately, although their delivery dates can delay testing.
QA Work Package | Person-Hours | Output Boundary |
|---|---|---|
Scope and test design | 8 | Reviewed selection and expected results |
Data and readiness checks | 6 | Usable fixtures and recorded environment checks |
Changed-behavior execution | 14 | Permission and retry evidence, including initial failure capture |
Targeted regression | 10 | Evidence for the selected adjacent flows |
Investigation and confirmation | 8 | Deeper failure analysis and agreed fix checks, excluding reruns already counted in regression |
Coordination and reporting | 4 | Plan updates, triage coordination, and completion handoff |
Total | 50 | Nonoverlapping QA effort under revision A assumptions |
Assume Maya and Leo can each contribute five net delivery hours per working day. Together that is ten person-hours per day, so 50 divided by 10 gives five capacity-days. This checks aggregate staffing capacity; it does not predict completion five working days after the start. Sequential dependencies, specialist access, and late corrected builds can extend elapsed duration.
Use availability after leave, support obligations, and unrelated meetings. Coordination already counted in the table consumes these delivery hours, so do not deduct it again. If you need a more detailed breakdown or estimate ranges, the approach to test effort estimation develops that forecast separately from the calendar schedule.
The proposed six-working-day window provides a review opportunity beyond the aggregate capacity calculation, not a guaranteed contingency allowance. Day labels below are relative working days; replace them with actual dates, availability, and review times in an operational plan.
Proposed Point | Work or Milestone | Prerequisite and Owner | Evidence Needed |
|---|---|---|---|
Day 1 | Confirm scope and start design/readiness work | Priya resolves acceptance questions; Nia begins setup | Reviewed scope, open-question owners, and readiness gaps |
Day 2 | Start high-risk permission and retry checks | Arun supplies candidate; Nia confirms access and reset | Candidate identity, smoke results, and first execution records |
Days 3-4 | Continue changed-flow checks and targeted regression | Maya and Leo use approved cases and stable data | Qualifying results, defects, and visible blocked work |
Day 5 | Investigate failures and confirm available fixes | Arun supplies corrected candidate and change notes | Confirmation evidence and affected regression assessment |
Day 6 | Review completion evidence and release recommendation | Maya reconciles results; Priya attends decision review | Dated summary, unmet criteria, and actual decision status |
Design and setup can overlap, and testers can investigate different flows in parallel. Confirmation cannot finish before the relevant fix exists. The table defines decision points rather than assigning an identical daily hour budget or promising that every dependency will arrive on time.
If the sandbox is unavailable on Day 2, Nia supplies a restoration forecast and Maya checks what unaffected work can continue. If a fix arrives too late for meaningful confirmation, move the review, revise scope through approval, or report the missing evidence. Adding a tester does not remove the need for a corrected build.
Define criteria while there is time to discuss them. Tie each condition to evidence and a responsible reviewer, rather than “QA is comfortable” or a copied pass percentage. Use the following as CP-43's proposed conditions, not universal thresholds.
A test effort can end because its time window expired without meeting completion criteria. Report that outcome as incomplete, with the exposure and next action, instead of quietly weakening the conditions. An exception request is still pending until the authorized person decides.
Keep release authorization separate. Even when the planned checks pass, the decision may depend on operational, security, commercial, or other evidence beyond this plan. Conversely, a release authority may accept a documented exception under the organization's process. Neither a pass rate nor a QA completion statement creates that authority or proves an absence of defects.
For each deliverable, state what must exist, who maintains it, and the point at which someone needs it. Avoid a list of document names with no connection to execution or decisions.
Before the entry check, Maya provides CP-43's reviewed scope, case or charter references, requirement revision, and ownership record. Nia supplies the environment/data readiness record; Arun supplies candidate identity and known limitations. Reference existing controlled records rather than duplicate their contents into several documents.
During execution, Maya and Leo maintain results identifying the case revision, candidate, environment, tester, outcome, and supporting evidence. Failure records need expected versus observed behavior and enough detail for investigation. Arun owns engineering responses; testers retain earlier failures when confirming corrections. Blocked and unrun work stays visible with an owner and next action.
At the completion review, Maya supplies a dated summary of assessed scope, qualifying results, open defects, missing evidence, and each completion criterion's status. Include a recommendation and distinguish it from Priya's actual decision, accepted residual risks, conditions, and timestamp. The test summary report template expands this evidence handoff. If no decision has occurred, record it as pending rather than filling a sign-off field in advance.
Suppose Priya introduces a second approver after testing starts. The permission combinations, fixtures, expected results, and potentially the release window have changed. Keep revision A and its review history, then create an identifiable revision with the reason and impact. Editing only the scope sentence leaves the old schedule and completion criteria misleading.
Continue the fictional example: 20 person-hours have been spent, and the owners now estimate 36 person-hours of remaining work after reviewing the new requirement and reusable evidence. The forecast becomes 20 plus 36, or 56 person-hours. These “actuals” are invented for the example. Remaining work is a fresh estimate, not the unspent portion of the original budget; the new total still does not determine a finish date by itself.
Use a short change procedure:
Review the plan after important defects, candidate changes, lost capacity, or dependency delays as well as requirement changes. Keep earlier evidence attached to its original candidate and expectations. A changed test should not make an old failure disappear.
Before committing to the window, ask a colleague to trace one high-risk refund behavior through its selected tests, owner, prerequisite, scheduled evidence, and completion condition. If that path ends in an assumption, resolve it or assign a named owner and review point. That is a more useful readiness check than whether every template heading has text beneath it.
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A green automated test history can help a developer identify a safe-looking change while leaving a QA lead unable to answer a different question:…
A sprint can contain carefully estimated testing work and still overload the only person who can perform it. The missing input may be a…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.