Flaky Test Quarantine: Owners, Expiry, and Nonblocking CI Results
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A password-reset link works during a demonstration. Someone calls that “QA complete.” Later, the team discovers that nobody agreed what should happen when the link is used again. The successful demonstration established something about the software, but it said little about how reliably the team defines and checks its requirements.
QA, QC, and testing describe related quality activities with different purposes. Understanding those purposes helps a team ask for the right evidence instead of treating “checked by QA” as a complete explanation. A single password-reset feature can show the differences without assigning quality to one department.
Quality assurance (QA) concerns the processes used to develop and evaluate software. Its purpose is to build justified confidence that quality requirements can be fulfilled through suitable practices, their consistent use, and improvement when they prove inadequate. Agreeing how requirements are reviewed, checking that reviews happen, and improving a weak review process are assurance activities.
Quality control (QC) focuses on evaluating work products against quality requirements. In software, those work products can include requirements, designs, source code, and a running application. The practical question is whether the thing being examined meets the criteria that apply to it, and what needs attention if it does not.
Testing is the set of activities used to discover defects and evaluate the quality of software work products. It includes analyzing what to test, designing checks, preparing conditions, examining artifacts, executing software where appropriate, and communicating results. Testing is a major form of quality control, but QC is broader than a synonym for test execution.
Terminology varies between quality frameworks and organizations. Some descriptions place QC within a broader QA system; others emphasize their distinct purposes within quality management. A useful working distinction is to identify the process being trusted, the work product being evaluated, and the evidence being produced. Avoid forcing every activity into an exclusive box.
Criterion | Quality Assurance | Quality Control | Testing |
|---|---|---|---|
Main Question | Can we trust how quality is built and checked? | Does this work product meet its quality requirements? | What do reviews and checks reveal about this work product? |
Password Reset Example | Define and improve the review and evidence process | Evaluate reset requirements, design, and behavior | Review token rules and check valid, expired, and reused links |
Typical Evidence | Review records and process improvement actions | Findings and conclusions against agreed criteria | Review findings, test results, and defect reports |
Timing | Across development, testing, and improvement | Whenever a relevant work product can be evaluated | Before execution through static work, and during execution through dynamic work |
The columns overlap deliberately. A test result can support a QC conclusion about the feature and give QA feedback about the process that produced it. The result does not become a different artifact because different people use it.
Consider a fictional account-management application. The following rules are illustrative assumptions for its password-reset feature, not a complete security specification or a report of tests performed on a real application:
The team must also agree what “expired” and “previously used” mean in its design. A tester cannot invent those rules from the interface and then claim that a matching implementation satisfies the business requirement. If the behavior remains undecided, the immediate task is clarification.
This feature supplies several possible work products: the written rules, the token-handling design, the code, the reset page, and the records of checks performed. Each can be evaluated at an appropriate point. Waiting for the finished page would miss opportunities to identify an ambiguous rule before it influences the implementation.
For this team, an assurance practice might require product, development, and testing representatives to review account-recovery rules before implementation. The review should address invalid states as well as successful recovery, record unresolved questions, and make the accepted behavior available to the people designing tests.
The assurance work includes deciding whether that practice is suitable and checking whether it is being used. A completed meeting invitation is weak evidence if the team cannot find the agreed token-reuse rule. A short review record containing the decision, its rationale, and outstanding questions is more useful than an elaborate checklist completed without discussion.
Assurance applies to the testing process too. The team might agree that a reset result needs a build identifier, relevant conditions, and an expected outcome before it can support a release discussion. Reviewing whether those records are usable is different from checking whether a particular reset attempt behaved correctly.
QA continues after coding. If reused-link defects recur, the team can investigate whether review prompts overlook state changes, whether requirement changes reach test authors, or whether evidence is being accepted from an obsolete build. Improving those practices addresses how the problem can be introduced or missed again. It does not remove the need to examine the current feature.
Quality control in software testing asks whether the current work product meets its agreed criteria. For the reset feature, a requirements review might find that the rule for a consumed token contradicts the design. A code review might identify a path that accepts a token without checking its consumed state. A check against a running build might reveal that the password changes even though the page reports rejection.
These findings concern specific artifacts or behavior. They can appear before the feature is complete, so QC should not be understood as an inspection reserved for the end of development. The criterion must also match the object: a complete requirement is useful, but its completeness does not establish that the implementation follows it.
The distinction between assurance and control becomes clearer when considering a review. Establishing a repeatable requirement-review practice supports assurance. Examining this feature's requirements and recording a contradiction evaluates a work product. The same review can contribute to both purposes without becoming two separate meetings.
QC conclusions need boundaries. “The reused link was rejected on the checked build” is narrower and more useful than “account recovery is secure.” The latter would require much broader evidence. This example does not establish the adequacy of the application's overall authentication design.
Testing provides much of the evidence needed for product evaluation. Some of that evidence comes from running software; some comes from examining work products without executing it. The ISTQB Foundation Level syllabus explicitly includes both static and dynamic testing, so reducing testing to clicking through a finished application leaves out important work.
Static testing examines a work product without executing the software under test. In the reset example, reviewing the requirements can reveal that “link becomes invalid after use” never states whether unsuccessful attempts consume it. That is a finding about an ambiguous requirement, even when no runnable feature exists.
A review of the implementation can likewise examine whether the token's consumed state is checked on the relevant path. The finding should identify the artifact and the issue clearly enough for another person to investigate. It should not claim an observed runtime failure when no execution occurred.
Calling every review “QA” hides this distinction. Checking the contents of a requirement or code change can be static testing. Checking whether the agreed review practice is consistently followed concerns process assurance. The object and purpose explain more than the meeting's name.
Dynamic testing executes software and compares observed behavior with expectations. For this feature, test analysis identifies relevant conditions such as valid, expired, and consumed links. Test design turns those conditions into checks with suitable account states, inputs, actions, and expected outcomes. Preparation makes the required build, environment, and data available.
For an unused valid link, a check might attempt an accepted password change and then verify the agreed resulting account behavior. For a consumed link, a separate check would attempt another reset and verify rejection and the unchanged password. Those checks answer different questions; a successful first reset cannot stand in for reuse coverage.
A useful result record connects the requirement and case version to the build and environment, expected and observed behavior, and any supporting evidence. Separate a planned check from a performed check. A case in the repository does not establish that it was executed, and an old passing result does not automatically describe the current build.
For a Jira-based team, Sherlock Test Management offers a relevant way to organize this distinction. Its product description separates a human-approved reusable test baseline from release-specific runs, keeps completed runs linked to the baseline version executed, and lets testers record Pass, Fail, or Blocked in Sherlock Runner or supported Jira panels. These are app-provided capabilities rather than Jira core features. Organizing cases and recorded results does not itself establish autonomous application execution, complete coverage, or authority to approve a release.
Suppose, within the fictional example, a consumed link permits another password change. This is an imagined outcome used to explain the workflow, not an executed test result. The tester records the expected rejection, the observed change, and the conditions needed to reproduce it. That evidence supports a QC finding against the agreed rule.
Debugging then investigates the cause and changes the implementation. Testing and debugging have different purposes: discovering or demonstrating a problem is different from diagnosing and fixing it. After the fix, confirmation testing checks the original failure condition again. Relevant regression testing examines whether the change affected other agreed behavior, such as a valid reset or rejection of an expired link.
The QA response asks a broader question: why did the development and testing process allow this omission? If the requirement review considered only the initial successful reset, the team can update its review prompts to include state changes and reuse. If a suitable case existed but was excluded without discussion, the improvement may concern scope review instead. The evidence should guide the process change rather than automatically adding another approval gate.
Passing the subsequent checks would support a bounded conclusion about the tested conditions. It would not prove that no defects remain. Unchecked conditions, unresolved findings, and environment limitations still belong in the release discussion, and the team's designated decision-maker must evaluate them against its release criteria.
A person called a QA engineer may perform testing, facilitate reviews, and improve processes. A developer writing a component test is also performing testing. A product representative resolving an ambiguous requirement contributes to quality without joining a QA department. Classify the activity by what it does, not by who happens to perform it.
For the fictional feature, the team could make the product representative responsible for clarifying recovery rules, developers responsible for implementation and technical checks, a tester responsible for investigating important user-facing conditions, and an engineering lead responsible for following through on agreed process improvements. It should separately name whoever can make the release decision. These are example assignments, not universal responsibilities attached to those titles.
Shared responsibility does not mean that every person has the same skills or that nobody owns the next action. It means quality concerns must be addressed across the work, while specific tasks still need accountable people and suitable review. Some contexts also require stronger independence than an informal team review provides.
Automation can assist repeatable checks, static analysis, and evidence collection. It cannot by itself decide whether the reset requirements describe the intended product, whether omitted scenarios are acceptable, or whether the quality process needs to change. More automated execution therefore does not remove the need for assurance or informed product evaluation.
Take a feature already under discussion and make its quality work explicit:
For the reset feature, the next useful action may be to resolve a missing reuse rule rather than rerun a successful demonstration. Asking for that specific decision produces a more meaningful quality conversation than asking whether “QA is done.”
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A green automated test history can help a developer identify a safe-looking change while leaving a QA lead unable to answer a different question:…
A sprint can contain carefully estimated testing work and still overload the only person who can perform it. The missing input may be a…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.