Flaky Test Quarantine: Owners, Expiry, and Nonblocking CI Results
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A refund change is nearly ready, and the release manager asks whether QA can finish this week. The tester can estimate the first execution pass, but the sandbox needs new data, permission rules are unresolved, and fixes will arrive after testing starts. Giving a single number now would hide the work most likely to change the answer.
Test effort estimation predicts the labor needed to meet agreed testing objectives. Effort is measured in person-hours or person-days. Duration is the elapsed working or calendar time between starting and finishing. A useful QA forecast keeps those quantities separate, exposes assumptions, and changes when the scope or evidence changes.
A work breakdown structure identifies what must be delivered. Three-point estimates expose uncertainty within that work. Together, they give a QA lead a forecast that stakeholders can question, revise, and use for a release decision.
Start with the release's changed behavior and the evidence needed to assess it. “Test refunds” is too broad to estimate consistently: one person may picture a happy-path check, while another includes permissions, retries, data preparation, and regression across payment states.
Consider an illustrative release that changes refund permissions and retry handling. The team agrees to test authorized and unauthorized refund requests, retry behavior after a timeout, and selected existing refund flows affected by the change. The forecast covers QA analysis, preparation, execution, defect investigation, confirmation testing, coordination, and reporting. Developer fix implementation, a new load-testing campaign, and a full payment-provider certification are excluded and owned separately.
Record the agreed acceptance criteria, application candidate, supported configurations, test levels and types, required coverage, and completion conditions. Name exclusions precisely enough that someone can recognize new scope. If a second approval step appears later, it changes the estimate rather than silently joining the original promise.
Use product risk to decide which evidence belongs in the scope before estimating its cost. Duplicate refunds and permission bypasses may justify deeper checks than an unchanged display label. Sherlock's explanation of risk-based testing and prioritization provides a method for making those scope decisions explicit. A risk ranking helps choose work; it does not convert directly into testing hours.
A work breakdown structure, or WBS, decomposes the agreed scope into smaller, manageable deliverables. PMI defines it as deliverable-oriented: the structure describes the work's outputs, while the schedule later describes sequence and timing. For a small release, keep the hierarchy compact enough to review with the people doing the work.
Under a parent such as “refund release test evidence,” create work packages for approved scope and test design, a ready environment and dataset, execution evidence, triaged defects and verified fixes, and the release test record. Split execution evidence into changed-feature checks and targeted regression when their scope or ownership differs. Split the release record into ongoing coordination and final reporting so neither disappears inside an execution allowance.
For each package, identify the activities needed to produce its output:
Assign one accountable owner to each package, even when several people contribute. For example, the QA lead owns scope and reporting; the environment owner supplies the sandbox; assigned testers own execution evidence. Estimate the labor included in this forecast, not everybody's entire involvement. The table below includes QA setup and readiness checks, but excludes infrastructure engineering and developer coding effort.
Define a completion condition before assigning hours. “Test data ready” might mean that the required payment states and user roles exist, their relationships are valid, and a reset has been demonstrated. Waiting for access is a dependency; diagnosing broken access is an activity. Recording both prevents a blocked day from being mistaken for a day of productive testing labor.
Check boundaries for duplication. Initial failure capture can belong to execution, while deeper investigation and later confirmation belong to defect work. If targeted regression includes a rerun after a fix, do not count that same rerun again in the confirmation package. The activities may overlap in time, but their estimated labor should not overlap in the total.
Ask the package owner for an optimistic estimate, a most likely estimate, and a pessimistic estimate, using the same scope, completion condition, and unit throughout. Denote them O, M, and P.
The optimistic scenario assumes favorable but credible conditions. The most likely scenario reflects the team's current judgment about normal work. The pessimistic scenario includes plausible adverse conditions, such as repairing a fixture or investigating repeated failures. It is not an allowance for every imaginable incident or unlimited new scope.
Have contributors form initial judgments before comparing them. Discuss large differences by examining assumptions: one tester may expect reusable cases, while another expects to redesign them. Compare with similar completed work when reliable records exist, adjusting for changed coverage, tooling, and experience. A historical execution average cannot account for newly required setup by itself.
For this example, use the weighted three-point calculation described in ISTQB's Foundation Level syllabus:
Expected effort E = (O + 4 × M + P) / 6.
This PERT-style estimate gives the most likely input greater weight. It is different from the triangular-distribution mean, E = (O + M + P) / 3, discussed in PMI's schedule-risk guidance. Choose the method deliberately and record it; changing the formula changes the result without changing any underlying work.
For environment and data preparation, assume O = 4, M = 8, and P = 18 person-hours. The weighted calculation is (4 + 4 × 8 + 18) / 6 = 9 person-hours. The triangular mean is (4 + 8 + 18) / 3 = 10 person-hours. Neither formula validates the inputs. The weighted result is a planning mean under the chosen model, not a deadline or a promise that the work will finish within a particular confidence interval.
The following figures are invented assumptions for the illustrative refund release, not measured productivity or a customer case. All entries are person-hours. The hierarchy separates changed-feature and regression evidence under package 3, and coordination and completion under package 5.
Work Package and Deliverable | O | M | P | Weighted E |
|---|---|---|---|---|
1 Approved scope and test design | 8 | 12 | 22 | 13 |
2 Ready environment and resettable data | 4 | 8 | 18 | 9 |
3.1 Feature execution evidence | 10 | 16 | 28 | 17 |
3.2 Targeted regression evidence | 6 | 10 | 20 | 11 |
4 Defect triage and fix confirmation | 4 | 8 | 18 | 9 |
5.1 Coordination and forecast control | 2 | 4 | 6 | 4 |
5.2 Completion report and handover | 2 | 4 | 6 | 4 |
Total | 36 | 62 | 118 | 67 |
The weighted planning total is 67 person-hours. The sum of the most likely inputs is 62 person-hours; these are different summaries of the same assumptions. The optimistic and pessimistic column sums, 36 and 118 person-hours, show the aggregated favorable and adverse scenarios. They are not empirically calibrated percentiles, and the pessimistic sum is not a maximum under every possible future change.
Make the assumptions behind the large ranges inspectable. The setup range assumes an existing sandbox and usable fixture tooling, with repair work possible. The execution ranges include initial failure capture but exclude deeper triage. The defect package includes QA investigation and confirmation of the anticipated fixes, with its adverse scenario allowing repeated clarification and confirmation. It excludes developer implementation and unlimited fix cycles. A late redesign requires a new estimate.
Add the weighted means because the effort packages do not duplicate work. This arithmetic does not require the packages to be independent. Independence becomes important when modeling the spread of the total: the same unstable sandbox or misunderstood requirement can increase several packages together. Do not shrink the forecast's uncertainty merely because the table contains several rows.
Capacity is a constraint on scheduling, not a substitute for it. Assume two testers can each contribute five net delivery hours per working day to the activities in this forecast. That gives ten team person-hours per day, and 67 / 10 = 6.7 capacity-days. This is an aggregate staffing check, not a forecast that testing will finish after 6.7 working days.
Use availability net of leave, support obligations, and unrelated meetings. Keep the coordination work already estimated in the WBS inside the available delivery hours, rather than deducting it again. Then assign the actual activities to people with the necessary skills and access.
In the example, a tester can develop cases while the environment owner prepares the sandbox, but execution waits for approved criteria, a testable build, and ready data. Only the tester with the required access may be able to prepare certain payment states. Confirmation testing waits for corrected builds, and the completion review waits for the relevant results. Adding another available tester does not remove those gates.
Write down prerequisite owners and promised readiness windows before proposing a finish date. The developer-to-QA handover agreement is useful here because it makes candidate identity, current criteria, setup, and recovery ownership checkable. A blocked sandbox can extend elapsed duration without consuming equivalent active QA effort if testers can work elsewhere; troubleshooting it can increase both. Record idle paid time separately if the budget also tracks the cost of reserved availability.
Review shared risks beside the schedule. A sandbox outage can affect setup, execution, and retesting together. Parallel paths that converge at release review also create completion risk that a simple sum of averages cannot describe. For a high-stakes deadline, assess the dependency network and uncertainty explicitly rather than attaching a confidence percentage to the PERT mean.
Keep ordinary rework in the relevant package ranges. If management requires a separate contingency reserve for an excluded risk, record the trigger, additional work or waiting time, owner, and approval rule. Check that it is not already represented in P. A universal percentage added for risks already represented in the package ranges can conceal double counting without explaining what the reserve buys.
Treat the forecast as a versioned decision record. Review it after test design, initial execution, a meaningful defect discovery, a readiness change, or revised acceptance criteria. The question is how much work remains under the current conditions, not how much of the original allowance remains unspent.
Continue the illustrative scenario: the team has spent 30 person-hours, has demonstrated the dataset reset, and now needs coverage for a second approver. After reviewing completed evidence and the changed requirement, the owners estimate the remaining work at O = 24, M = 38, and P = 64 person-hours.
The remaining weighted estimate is (24 + 4 × 38 + 64) / 6 = 40 person-hours. Forecast total effort is therefore 30 actual + 40 remaining = 70 person-hours. The remaining estimate includes the new approval checks and excludes completed work. It is a fresh judgment, not the old total with actual hours subtracted.
Use a repeatable update procedure:
Do not infer remaining effort directly from a pass count. Completed simple checks may leave a difficult integration and defect-confirmation work untouched. Track the work needed to obtain acceptable evidence, including failures and blockers, rather than treating every case as an interchangeable unit.
Jira Cloud can hold native time estimates and track time spent and remaining estimates; configuration differs between company-managed and team-managed setups. Keep the O/M/P rationale and scope version in an agreed forecast record or deliberately configured fields. Do not assume a native time field automatically calculates a three-point forecast, or convert story points into hours without a separately justified model.
For teams managing release evidence in Jira, Sherlock Test Management describes human-approved versioned test baselines, release-specific scope, recorded results, and linked defects. Those records can help reviewers identify changed coverage and unfinished investigation before re-estimating. Proposed case updates still require human approval. This is a source of reviewed scope and execution evidence, not a claim that Sherlock automatically predicts labor or release dates.
The QA lead should own the consolidated forecast, while package owners defend their assumptions and update remaining work. Development and environment owners commit to prerequisite readiness. The product or release owner decides whether to change scope, supply capacity, move the proposed date, or accept explicitly recorded residual product risk. Accepting a labor range and accepting release risk are separate decisions.
Before using the forecast for a commitment, put the scope version, person-hour ranges, weighted planning total, exclusions, dependency windows, capacity assumptions, and next review trigger in front of those owners. Ask them to challenge the widest ranges and the least reliable prerequisites first. If the requested date cannot accommodate the agreed evidence, revise the plan openly instead of compressing the numbers until they fit.
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A green automated test history can help a developer identify a safe-looking change while leaving a QA lead unable to answer a different question:…
A sprint can contain carefully estimated testing work and still overload the only person who can perform it. The missing input may be a…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.