Flaky Test Detection: Use CI History Without Hiding Real Regressions

A checkout test fails because the confirmation page never appears. Its retry passes, and CI finishes green. The team still needs to know whether the first attempt exposed a broken checkout, a test synchronization error, shared data, or an unavailable dependency. The green result answers none of those questions.

Flaky test detection uses execution history to identify inconsistent outcomes under comparable conditions. It should preserve the original regression signal while directing investigation. A fail-then-pass sequence is evidence of inconsistency, not permission to dismiss the failure. Use the following workflow to examine existing CI history and turn a candidate into an evidence-backed decision.

Preserve the Failure Before Interpreting the Retry

Playwright Test distinguishes tests that pass immediately from tests that pass after retry. For ordinary tests expected to pass, its retry documentation describes three categories: passed, flaky, and failed. Keep those categories separate in your history instead of storing only the eventual successful attempt.

Observed Sequence	Initial Interpretation	Required Follow-Up
Pass on the first attempt	Passed in this execution	Retain context for comparison; one pass does not establish reliability
Fail, then pass on retry	Flaky under Playwright's retry classification	Preserve the first failure and investigate why outcomes differ
Fail on the first attempt and every retry	Failed in this execution	Investigate the failure without assuming either regression or test defect
Missing, interrupted, or skipped execution	Incomplete evidence for the required behavior	Recover the missing context or obtain the required execution evidence

These outcome labels do not identify the faulty component. An application race can produce a genuine user-visible defect intermittently. A broken locator can also fail consistently. Treat the runner outcome and the investigation cause as separate fields so that neither overwrites the other.

Retries also change execution context. After a failure, Playwright discards the worker process and browser; a new worker can rerun setup hooks before retrying the test. A successful retry may therefore follow different initialization or timing. It does not demonstrate that the original failure was harmless.

Account for deliberate annotations too. A test expected to fail can behave as expected while its raw attempt status is failed. In serial groups, one failure can skip subsequent tests, and the group is retried together. Preserve expected status and grouping rather than treating every raw failure or skipped record as an independent flaky test.

Build a History That Keeps Comparable Attempts Together

Start by checking whether the retained reports can answer two questions: which behavior ran, and under which conditions? A long history is misleading when it merges different tests or loses the context that made their outcomes different.

Separate Test Identity From Execution Identity

Use a logical test identity that includes the repository, test file, full title path, project or browser variant, and relevant parameter values. Retain the framework's identifier as well. Playwright documents its test ID as derived from the file, title, and project, with uniqueness within a session. Do not assume that an identifier remains unchanged through every rename or reorganization.

When a test moves or changes its meaning, review the mapping between old and new identities. Two tests named “checkout succeeds” in different files are not necessarily the same check. Conversely, assigning a fresh random identity to every execution prevents useful historical comparison.

Keep the following context alongside each execution:

• Source revision actually checked out, application build or image deployed, and test-code revision when separate.
• Framework and browser versions, project settings, feature flags, timeouts, retry policy, and relevant dependency configuration.
• Environment identifier, operating system or runner image, dataset or fixture version, and controlled clock or random seed where applicable.
• CI workflow and job identity, workflow attempt, shard, worker, test retry ordinal, and timestamps.

Group outcomes first by test variant and relevant code, deployment, and configuration context. Keep attempt, shard, and worker identifiers attached to the observations. Do not make every worker or retry its own test identity, which would split related attempts into isolated records. Compare those execution dimensions when looking for correlations.

The same source commit is useful evidence, but it is not a complete environment description. A shared staging deployment, mutable container tag, feature flag, or backend dataset may change without a new commit in the test repository. Record those differences before describing two results as equivalent.

Keep the Original Evidence and Detect Missing Reports

For every attempt, retain the observed and expected status, retry number, start time, duration, error details, and available attachments or diagnostic logs. Playwright's reporter APIs expose individual results and attempt-level information. Its JSON and blob reporters provide useful machine-readable records; blob reports also support merging sharded results. Verify what your actual export and downstream parser retain rather than assuming a converted report preserves every retry.

Store artifacts under execution-specific identities so that a later rerun cannot overwrite the earlier report. On GitHub Actions, workflow reruns keep the run ID while the workflow attempt number increments. A workflow attempt is separate from a Playwright test retry. Retain both, together with job and shard identity, when matching reports to executions.

Check capture coverage before relying on a trace. Playwright's on-first-retry setting records the first retry, not the original failed attempt. That trace can show a passing retry while the first failure has only an error and logs. A policy that retains traces from failed attempts can preserve different evidence, at a collection and storage cost. Choose capture deliberately, protect sensitive data in reports, and retain artifacts long enough for the agreed investigation window.

Finally, reconcile expected report producers against received reports. An absent shard, cancelled job, expired attachment, or parser error leaves a gap. Record that gap explicitly. An empty result set must not become “zero failures,” and missing required executions must not become a passing release assessment.

Read the History Without Mixing Denominators

Consider an illustrative checkout-confirmation test with ten comparable initial executions. Assume the same source commit, deployed build, browser variant, configuration, and dataset definition, with an isolated starting copy for each execution. The sequences are:

Pass; pass; fail then pass; pass; fail then fail; pass; fail then pass; pass; pass; pass.

This invented history contains ten initial executions and thirteen total attempts. Three executions failed initially, two recovered on retry, and one still failed after retry. Report those observations separately:

• Initial-failure frequency: 3 of 10 executions, or 30%.
• Retry-recovery frequency: 2 of 10 executions, or 20%.
• Persistent-failure frequency: 1 of 10 executions, or 10%.

These are explanatory counts, not a benchmark or an estimate of production defect frequency. Reporting only the final failure would conceal two initial failures. Pooling every attempt into a single denominator would answer another question because retries are selected after failure and can run under changed conditions. They are not an independent random sample of ordinary executions.

Keep the sample size, observation window, retry policy, and incomplete executions visible beside any rate. A change in retry settings can alter the recovered category without fixing the underlying problem. A quieter week may reflect fewer executions rather than better reliability.

Separate configurations before ranking candidates. Failures confined to one browser or environment may be repeatable within that variant even when the combined history looks inconsistent. Within each cohort, compare failure signatures, such as the failed assertion and relevant error context. A historically flaky checkout test can reveal a new payment regression; an old label must not suppress a new signature.

Investigate the Candidate Before Assigning a Cause

A mixed-outcome record should open an investigation with an owner and a question. Begin with “suspected intermittent failure” or an equivalent local disposition. Keep the raw runner outcome intact while recording hypotheses separately.

1. Inspect the earliest failed attempt. Identify the failed assertion, expected behavior, actual observation, and the requirement supporting the expectation. A timeout says an expected condition was not observed in time; it does not identify which component caused the delay.
2. Check comparison validity. Confirm the actual test revision, deployed application build, browser, settings, fixture definition, and dependency context. Separate a pass after a deployment change from a pass on the same controlled setup.
3. Reconstruct the starting state. Preserve relevant redacted records, request identifiers, timestamps, and setup results before cleanup removes them. If reconstruction is impossible, retain that limitation instead of claiming a confirmed cause.
4. Plan controlled reproduction. First preserve the original context, then vary one suspected factor at a time, such as worker concurrency, test order, data isolation, or a dependency response. Record which hypothesis each comparison tests and retain failed as well as successful attempts.
5. Review the explanation against the observed failure. Assign a supported cause, or leave the result inconclusive with the missing evidence and next action specified. Repeated clean runs alone do not prove the original defect cannot recur.

Data reset deserves particular attention. A fresh browser or worker does not restore a consumed order, reserved account, or server-side cache. Use a versioned fixture definition and verify its starting conditions rather than rerunning against whichever records remain. The test data management guide develops the dataset contracts, isolation, and reset controls needed for that comparison.

For the checkout example, several explanations remain possible. An assertion that samples the page before the expected transition may need a condition-based wait. Shared account data may let another test invalidate the order. A runner outage may prevent the application from being reached. The application itself may acknowledge an order before its required state is committed. Each hypothesis calls for different evidence and a different owner.

Do not classify an error as infrastructure noise merely because many tests fail together. A broken application dependency or deployment can produce the same cluster. Likewise, success with one worker suggests a concurrency relationship but does not establish whether the defect belongs to the test fixture or the application. Preserve the customer-visible expectation while narrowing the explanation.

Set a Gate and a Deadline for Containment

Decide separately how an unresolved candidate affects merging or releasing. Detection records inconsistency; gating expresses the evidence and risk the team is willing to accept. A final green job should not silently make that policy decision.

Playwright documents a failOnFlakyTests option, added in version 1.52, that makes marked-flaky tests cause an error exit. Where the installed version supports it, a team can consider that policy to keep retry recoveries blocking. It improves visibility but can also hold unrelated work while investigation proceeds. Evaluate it against the required behaviors and the team's response capacity rather than treating it as a universal setting.

If temporary quarantine is justified, record the precise test variant and failure signature covered, why it is being removed from the blocking gate, and which behavior loses that protection. Continue collecting its results where feasible. New signatures, broader impact, or a changed requirement should trigger reassessment rather than inherit the exception automatically.

Give the exception a named owner, investigation deadline, review date, expiry behavior, and restoration criteria. Identify compensating coverage or explicit acceptance of the remaining exposure. A critical payment assertion with no equivalent check may need to remain blocking despite the cost. The defect triage workflow explains how to turn uncertain findings into accepted next actions, impact decisions, and review commitments.

Restoration should require more than one successful rerun. Review the correction against the original failure, verify the relevant conditions and broader impact, and examine a pre-agreed observation window with its execution count and configurations. There is no universal number of clean runs that proves reliability. If the deadline expires without adequate evidence, escalate the missing coverage instead of silently extending the exception.

Start With One Unresolved Failure

Choose one recent fail-then-pass result whose cause is still unknown. Recover its first-failure evidence, confirm which executions are genuinely comparable, and assign the next investigation action to a person who accepts it. Keep the failure visible until its explanation and release treatment are reviewed. CI history is useful when it preserves what happened and helps the team decide what must happen next.