Flaky Test Quarantine: Owners, Expiry, and Nonblocking CI Results
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A team can have a large test repository and a busy execution dashboard while still being unable to explain why particular risks received attention or who can accept the remaining uncertainty. Test management addresses those decisions and the coordination needed to carry them out.
Test management is the discipline of deciding and coordinating how testing will meet its objectives within the available resources, risks, and organizational constraints. It connects the purpose of testing with its scope, priorities, people, evidence, and accountable decisions. In software test management, the object being assessed may be an application, a component, a change, or another software work product.
The discipline helps people choose and organize useful testing, respond when circumstances change, and explain what the evidence supports. It cannot guarantee a defect-free product or give someone release authority that the organization has not assigned.
Testing has to answer a purpose beyond completing a collection of cases. Stakeholders may need evidence about whether a change meets agreed requirements, whether a significant product risk has been investigated, or where an unresolved problem needs attention. Management translates those needs into a testing approach that people can understand and carry out.
Its goals are connected:
These goals require choices. Expanding scope creates additional work; keeping scope narrow can leave important uncertainty unresolved. A request to finish earlier may require a different approach, more suitable resources, less scope, or explicit acceptance of a remaining risk. The management responsibility is to expose the consequences and obtain the relevant decisions, rather than quietly changing the meaning of completion.
A feasible plan also needs more than an estimate. Someone must confirm whether the required skills and resources are available, whether dependencies are ready, and whether proposed responsibilities have been accepted. A named assignee in a draft does not prove any of those conditions. Where the facts are missing, management should preserve the unknown and seek clarification instead of publishing a confident schedule.
Testing everything is generally infeasible outside trivial cases. Good test management therefore makes the basis for prioritization visible. It does not treat a large case count, an attractive dashboard, or an ambition to achieve complete testing as proof that the important questions have been answered.
Management activities include planning, monitoring, control, coordination, reporting, and decisions within an assigned mandate. Planning establishes objectives, approach, scope, resources, and criteria. Monitoring gathers information about what is happening. Control uses that information to respond to deviations, changed risks, or blockers. These activities can continue while testing work is underway; they need not form a separate sequence of phases.
The testing work itself includes analyzing the test basis, designing checks, preparing test data and other test materials, implementing tests, executing them, comparing actual and expected behavior, and recording results. These activities create the evidence that management needs. Coordinating their scope and priorities does not mean that every design choice or test execution becomes a management activity.
Consider the difference in the questions being answered. A tester asks which conditions to examine and how to recognize an incorrect result. The management role asks whether the chosen work addresses the agreed objective, what prevents it from proceeding, and which decision or resource is needed next. One person may answer both kinds of question, but the distinction keeps evidence creation separate from decisions about the overall effort.
A dedicated test manager can take responsibility for the overall testing effort, lead the team, and coordinate with stakeholders. In a smaller or Agile team, management responsibilities may be distributed among people who also design or execute tests. ISTQB describes role allocation as dependent on project and product context, people's skills, and the organization. The activity does not disappear because there is no separate management position.
Shared responsibility still needs explicit decision rights. The team should know who can approve test scope, who coordinates unresolved dependencies, who maintains the testing view of risk, and who receives an escalation when the proposed work is not feasible. Contributors can supply information without owning every resulting decision. Likewise, assigning someone to coordinate testing does not automatically make that person responsible for fixing the implementation or approving the business outcome.
Useful accountability is specific enough to resolve a disagreement. “The team owns quality” expresses a shared concern, but it does not identify who can accept a proposed scope reduction. Record that authority where it is agreed, and keep proposals distinct from accepted commitments.
Testers supply findings and other evidence. The management role synthesizes that information, reports limitations, adjusts work within its mandate, and recommends further action where needed. It may also have assigned authority over particular testing decisions. None of those responsibilities automatically makes a test manager the business acceptance authority or the release decision owner.
Test completion, business acceptance, and permission to release are related but different decisions. Completing an agreed testing activity does not establish that every business concern has been resolved. A release decision may need other evidence and approvals as well. Organizations should make the relevant authorities explicit, including who can accept remaining risks, rather than infer them from a job title or a report's status.
Progress, pass rate, coverage, and remaining risk answer different questions. Progress describes work against a stated plan. A pass rate describes the share of a stated result population recorded as passed; its interpretation depends on which results are included and how blocked or unexecuted work is treated. Neither tells the reader, on its own, whether the selected tests addressed the right concerns.
Coverage describes a relationship to a stated basis. Requirements coverage, risk coverage, and code coverage use different populations and should not be substituted for one another. Even a recorded link between a requirement and a case does not establish that the case was executed against the relevant version or that its checks were adequate. A management report needs enough context to distinguish planned coverage from the evidence actually obtained.
Remaining, or residual, risk concerns what could still go wrong after considering that evidence. Known defects, untested behavior, blocked investigations, changed assumptions, and limitations of the test approach can all matter. A favorable pass rate cannot establish that these concerns are acceptable. Testing can reveal defects, but finding none does not prove their absence.
Traceability helps connect the reasoning: the objective or requirement, the relevant risk, the test work, the observed result, and any defect or unresolved question. Management can then explain why a gap matters and whether the response should be further investigation, a resource request, a revised proposal, or an escalation. The report informs a decision; it is not permission to release.
The following brief shows the boundary between coordinating testing and carrying it out. Its identifier is an illustration reference, not a field in a test management product. The example deliberately leaves missing prerequisites unresolved rather than inventing people, results, or approvals.
MGT-24.1 is a fictional, unexecuted management decision brief for a possible future change to delivery-address selection at checkout. Its bounded objective is to obtain evidence about the changed selection behavior and its effect on order totals, so an authorized stakeholder could later judge the remaining risk. A draft change description names address selection as changed and order totals as a potentially affected dependency. Incorrect delivery details and incorrect totals are candidate risks, not observed failures. No approved acceptance rules, implementation version, test results, or validated coverage are supplied.
The proposed testing boundary is the changed address-selection behavior and the potentially affected totals dependency. Other checkout behavior has not been assessed for inclusion or exclusion; the proposed boundary is not an approved scope baseline. Available tester capacity, required skills, environment readiness, test data permissions, and dependency readiness are Unknown. No dates, effort estimates, named owners, or accepted assignments are supplied. The brief authorizes no application access, data upload, test execution, record change, or automated action.
The business acceptance authority, release decision owner, and person authorized to approve test scope are Unknown. Applicable acceptance criteria, test entry and exit criteria, and any mandatory organizational obligations are also Unknown. Neither a testing recommendation nor a future passing result would itself authorize release. Actual execution and acceptance are Unknown / NOT RUN; no management proposal in this brief is an accepted commitment.
The proposed management response is to put the changed selection behavior and totals dependency first in a scope discussion because they connect the draft change to the candidate delivery and financial risks. A proposed coordination role would request scope approval, capacity and skill confirmation, environment and dependency readiness evidence, and permission for any required test data. These requests are recommendations only. No person has accepted the role or the work, and no schedule or feasible capacity allocation has been established.
Testing work, if separately authorized and ready to start, would analyze the approved acceptance rules, design relevant checks, prepare permitted data, execute those checks, and record actual outcomes with their tested version and conditions. That work has not run. The management role would use the resulting evidence to revisit priorities, report blockers and coverage gaps, and make recommendations within its assigned mandate. It would not invent expected behavior from the candidate risks or convert a proposal into an approved assignment.
The next management decision should remain a request for clarification and approval, not a declaration of readiness. Scope approval requires an identified scope authority and an accepted boundary; permission to start testing also requires the applicable entry criteria and authorized resources, environment, dependencies, and data to be established. Test completion would need the applicable exit criteria and evidence to be assessed. Business acceptance and release would remain separate decisions for explicitly assigned authorities. All actual approvals, commitments, execution results, and acceptance outcomes remain Unknown / NOT RUN.
Decision | Proposed Management Response | Testing Work and Authority Boundary |
|---|---|---|
Priority and scope | Discuss changed address selection and the totals dependency first; request approval of the boundary. | Analyze approved acceptance rules before designing checks. Other checkout scope remains unassessed. |
Readiness and coordination | Request capacity, skills, environment, dependency, and test data permission evidence. | No accepted assignment or execution exists. Starting work requires established entry criteria and authorization. |
Evidence and control | Revisit priorities and report blockers and coverage gaps when actual evidence becomes available. | Record outcomes against the tested version and conditions. Candidate risks do not define expected behavior. |
Completion and release | Request clarification of exit criteria and the separate decision authorities. | Test completion needs criteria and evidence assessment. Passing tests alone do not authorize business acceptance or release. |
The proposed responses preserve the distinction between organizing work, producing evidence, and granting permission. They do not claim that the checkout change is correct or that the suggested scope is sufficient. Resolving the missing criteria and authority could change the proposal.
A test management tool can help organize test materials, connect related records, gather progress information, and support collaboration. A repository makes cases retrievable; traceability makes relationships inspectable; reporting makes recorded information easier to discuss. These capabilities are useful when the records remain current and their meaning is understood.
The tool does not decide whether the objective is appropriate, whether an untested dependency matters, or whether someone has authority to accept a risk. A populated repository is not evidence of execution, and a dashboard cannot make missing results appear. Teams still need to distinguish proposed work, approved scope, actual attempts, and reviewed outcomes.
Management tools, test design tools, execution tools, and collaboration tools can support different activities. A capability in one category does not establish support for every other category. In particular, organizing or recording testing does not itself demonstrate autonomous test execution, an AI-driven launch, or a particular integration. Those claims require their own verified scope and prerequisites.
When testing becomes difficult to coordinate, identify the decision that is preventing useful progress. State the objective, the evidence already available, the important unknowns, and the authority needed to resolve them. Then make a bounded proposal that the relevant people can review and accept. That gives testing a clear direction without confusing a plan with completed work or a recommendation with permission.
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A green automated test history can help a developer identify a safe-looking change while leaving a QA lead unable to answer a different question:…
A sprint can contain carefully estimated testing work and still overload the only person who can perform it. The missing input may be a…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.