A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees what the test does, owns the repair and decides when its result can become blocking again. Otherwise, a quieter pipeline conceals an unresolved coverage gap.
Flaky test quarantine is temporary containment: limit the effect of an already-reviewed unreliable result while preserving the test and its evidence. It does not repair the test. Start with an established classification and retained failure context, not the assumption that passing on one retry proves the original failure was harmless.
Review the original failure, affected code and test versions, branches, environments and protected behavior before approving an exception. A flaky test can still expose a genuine product problem. If the observed failure is a stable defect, or its cause remains too uncertain to justify containment, investigate it rather than quietly removing its gate.
Separate the accountable owner from the person editing a manifest or operating CI. The owner accepts responsibility for investigation, evidence, review and follow-through. Decision authority belongs to the role authorized to accept the coverage risk. A maintainer's permission to change a pipeline does not automatically grant that authority.
Before admission, record:
These are recommended team-record fields, not a promise that every test platform provides matching controls. Link the record to its change history so a renamed test or modified assertion does not silently inherit an unrelated exception. For security-critical checks and mandatory release gates, require the applicable risk decision and credible compensating evidence. If that decision is unavailable, keep admission blocked.
A nonblocking test should still produce a result. Buildkite Test Engine distinguishes muted tests, which continue running without their failures affecting the build result, from skipped tests, which do not produce execution data. Those are documented platform meanings. In another integration, inspect the runner behavior rather than assuming that a label named quarantine changes execution or exit status.
Buildkite documents a supported route through its Test Engine Client, bktec. Test-state management has plan and suite-permission prerequisites, and support for muting and skipping varies by framework. Its current client support table includes both for RSpec; it does not establish identical support for every runner or installed client version.
For that RSpec route, an authorized maintainer would first confirm the suite configuration and authentication, select the rspec runner, set the result path and preserve the required JSON result output. The documented custom-command requirements include the JSON formatter and result-path placeholder. Test identifiers and file paths must match the recorded suite data; when a collector uses a location prefix, the runner setup must account for it.
Result delivery needs a separate check. The current bktec README describes built-in uploads as off by default and available from version 2.7.0. Choose one supported upload route, either that configured uploader or an appropriate collector, rather than uploading the same run twice. Verify the deployed version and runner-specific instructions before adopting the route. No client installation, configuration change or test run is performed here.
If bktec is not used, Buildkite describes querying test states through its API and configuring build scripts accordingly. That places implementation responsibility on those scripts; the metadata alone does not isolate failures. Review which results can be nonblocking and which errors must still fail the job. Turning every exit status into success, or applying allow_failure globally, can hide configuration failures, infrastructure incidents and unrelated defects along with the intended exception.
GitLab illustrates another boundary: its JUnit unit test reports display results but do not determine job status. The job script's exit status still controls failure. Report collection therefore does not implement safe quarantine by itself. GitLab's handbook also describes its own internal quarantine process, including ownership and monitored dequarantine. That internal QA workflow is not a universal GitLab CI feature, and some of its quarantine paths remove tests from CI execution rather than preserving muted execution.
Keep build and code identity, test version, environment, initial failures, retry attempts, per-test outcomes and diagnostic artifacts accessible through the quarantine record. Preserve the distinction between an initial failure and a later retry pass. A green overall build means that the configured gates allowed it to continue; it does not mean that the muted test passed or that its repair was verified.
Reconcile the expected quarantined-test membership with the results actually received. A test may be filtered out, a job may receive no assigned tests, or a report may be missing, expired or malformed. None supplies a passing observation. GitLab's report documentation also warns that duplicate test names can hide results, so counts alone cannot establish that the intended test was observed.
If safety requires a test not to run, record the coverage gap and the reason for skipping it. A separately approved diagnostic run in a safe context may provide future evidence, but planning that work is different from creating a scheduled job. Do not claim execution history for the interval in which the test did not run. Keep infrastructure and reporting failures visible, with an explicit blocking or escalation rule, instead of letting them disappear under a nonblocking-test label.
An expiry is a deadline for reconsidering the exception, not evidence that the test has recovered. Agree who reviews it, which clock and timezone apply, what happens exactly at the boundary and who receives an unresolved decision. A date without accepted responsibility can leave a test muted long after its original owner has moved on.
At review, examine the latest attempts, repair progress, affected coverage and the reason the exception still exists. A renewal should be explicit, authorized and bounded by a new expiry. If renewal is not justified, hold the affected coverage decision and escalate it. Do not automatically delete the test, discard its failure history or restore a known flaky gate merely because time elapsed.
GitLab's internal handbook uses bounded phases and its own cleanup process. Treat those as an organizational implementation, not a universal expiry duration or permission to remove coverage. Your policy must define the consequence that fits your release authority and risk model.
Require a repair that addresses the recorded failure mode. Then review evidence from the repaired version in the contexts implicated by that failure, which may include environment, ordering, shared state or parallel execution. A convenient isolated pass does not cover an unexamined context.
Agree repeat and observation criteria before judging the repair. Their scope should reflect the failure mechanism and the opportunity to observe it; there is no universal number of green runs that proves reliability. Retained retry history helps explain behavior, but it does not establish that nondeterminism has been eliminated.
An authorized return decision should name the repaired version, retained evidence and accepted monitoring plan. Specify an accountable observer, what recurrence requires and who may change the gate again. After the approved return, failures should again affect the blocking suite as intended. Monitoring cannot become an undeclared extension of nonblocking quarantine, and recurrence should not silently authorize another exception.
The following QP-v1 policy and Q-EXAMPLE-v1 record form one fictional illustration. They are authored team conventions, not vendor product fields, observed CI output or a deployable configuration. The confirmed-flakiness statement is a supplied assumption; all operational evidence and approvals remain as stated in the record.
Policy QP-v1 is a fictional, unexecuted team convention, not a vendor feature. Admission requires an already-confirmed flaky classification with retained failure evidence, a stable test identifier and suite, the test definition version, affected branches and environments, a reason, a repair reference, an accountable owner who has accepted responsibility, and an authorized risk decision. The decision must define the exact containment scope and compensating coverage. Stable defects, security-critical checks and mandatory release gates cannot be silently admitted. Missing ownership, context, evidence or decision authority blocks admission.
Containment requires the selected test to keep running in the approved context while only its reviewed failure category is nonblocking. Preserve code and build identity, environment, every initial and retry attempt, per-test result and diagnostic artifacts in accessible history. Reconcile expected tests against received results. A skipped or absent test, a missing report, or an infrastructure failure is not a passing test. Infrastructure failures and failures outside the approved exception remain visible and blocking under this policy. Unknown runner support, result delivery or failure isolation blocks readiness; an overall green build does not change the test outcome.
Before admission, the owner and decision authority must agree an explicit expiry timestamp in UTC, a review boundary and an owned escalation route. Under QP-v1, the exception is expired when the review clock is equal to or later than that timestamp. At expiry, review the evidence and either record an authorized, bounded renewal with a new expiry and reason, or place affected coverage on hold pending a decision. Expiry does not mark the test repaired, erase history, delete the test or blindly restore a known flaky gate. Missing expiry or review responsibility blocks admission.
Return to blocking requires a repair linked to the original failure mode, verification on the repaired version in all affected contexts, reviewed repeat criteria and retained attempt history, an authorized return decision, and an accepted monitoring plan with an accountable observer and recurrence response. A retry pass alone is insufficient, and no universal number of passes proves reliability. Unknown verification, context coverage, criteria, approval or monitoring blocks return.
After an approved return, observe the test against the agreed monitoring criteria and retain new results alongside quarantine history. A recurrence requires retaining the new evidence, referring the case to the owner and decision authority, and reassessing risk and coverage. QP-v1 grants no automatic permission to re-quarantine or to leave the test permanently nonblocking. Closure requires reviewed repair evidence, completion of monitoring and an authorized decision; unknown outcomes keep closure blocked.
Record Q-EXAMPLE-v1 proposes containment for test report_export_selected_rows in suite Reporting, definition TEST-v1, branch maintenance-demo and environment staging-demo. Confirmed flakiness is a supplied fictional assumption, not a detection result. The proposed reason is intermittent report-export checking; its cause remains Unknown. Repair reference REPAIR-DEMO is a fictional placeholder, not a created issue. No actual failure evidence, code or build identity, attempts or artifacts have been supplied.
Owner acceptance, decision authority, risk review, compensating coverage, expiry timestamp, review responsibility and escalation route are Unknown. Runner support, result delivery and failure isolation are Unknown. Repair, context verification, repeat criteria, return approval, observer acceptance and monitoring outcomes are Unknown. Actual quarantine execution, repair verification and return are NOT RUN. The record is not admitted, is not launch-ready, cannot return to blocking and cannot be closed. No Buildkite, GitLab, Jira or CI action, test execution, muting, skipping or schedule was performed.
Lifecycle Decision | QP-v1 Requirement | Q-EXAMPLE-v1 Evidence | Current Decision |
|---|---|---|---|
Admit quarantine | Confirmed classification, retained evidence, exact context, accepted owner, risk authority, scope and compensating coverage | Flakiness is assumed; evidence, owner acceptance and authority are Unknown | Admission blocked |
Start nonblocking execution | Supported runner, approved failure isolation, continued execution, retained attempts and reconciled reports | Runner support, failure isolation and result delivery are Unknown; execution is NOT RUN | Readiness blocked |
Review expiry | Explicit UTC expiry, review boundary, responsible owner and escalation route | Expiry and review responsibility are Unknown | Admission blocked; no expiry assessment |
Return to blocking | Verified repair in affected contexts, reviewed repeat criteria, retained history, authorized return and accepted monitoring | Repair verification is NOT RUN; criteria, approval and monitoring acceptance are Unknown | Return blocked |
Close or handle recurrence | Completed monitoring and authorized closure; retain recurrence evidence and refer for a new decision | Monitoring outcomes are Unknown; no recurrence observation was supplied | Closure blocked; recurrence response unexecuted |
The table preserves the current uncertainty. A named placeholder is not an accepted owner, REPAIR-DEMO is not an implemented repair, and assumed flakiness does not supply failure artifacts. Because no expiry timestamp exists, the record cannot even support an expiry assessment. Its coverage gap must remain visible rather than being converted into a fictional green result.
The proposed nonblocking route also remains unready until continued execution, selective failure handling and complete result delivery are verified through an authorized process. Nothing in this illustration starts those actions. Its repair and return conditions are conjunctive: satisfying one while leaving approval, affected contexts or monitoring unknown does not permit dequarantine.
For a real quarantine candidate, bring the owner and decision authority a record that makes the next decision possible: what evidence exists, which behavior has lost blocking protection, what remains unknown and which repair or review action is due. If a prerequisite is missing, keep that boundary explicit instead of relying on the build's overall color.
The useful endpoint is an observed, approved return to meaningful blocking coverage, with the original history still available. Until then, treat quarantine as owned maintenance work with a visible coverage cost, not a completed fix.
A tester reviewing a Jira issue may need a fresh smoke run without opening GitLab and choosing pipeline parameters. A manually triggered Jira Cloud…
A regression-results worker receives HTTP 429 while reading Jira issues. Sleeping that worker may be necessary, but it will not help if other workers…
A checkout test times out while waiting for an order confirmation. The runner reports an assertion failure, the application records a failed request, and…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.
