Flaky Test Quarantine: Owners, Expiry, and Nonblocking CI Results
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A registration endpoint returns success and saves an account, yet the person signing up never reaches an authenticated session. The verification message might contain the wrong link, the activation request might update the wrong record, or the browser might lose its session during a redirect. Passing one connection does not establish that onboarding works as a journey.
Integration testing checks selected connections between components. End-to-end testing checks whether a declared workflow reaches its intended outcome across the assembled application. Use integration tests to examine the handoffs in detail and E2E tests to verify that those handoffs support a completed task. The useful comparison is what each test can demonstrate, which dependencies it needs, and how much uncertainty remains when it fails.
Testing terminology varies. Some teams call narrowly scoped adapter checks against doubles integration tests; others use the same term for several running services. End-to-end and system testing also overlap in some taxonomies. Agree on the boundary before interpreting a test name.
Here, an integration test exercises selected real collaborating components, potentially with controlled dependencies outside that boundary. It can include more than a pair of components. An E2E test follows a complete selected journey through the declared system boundary. Neither label means every production service must be running.
Criterion | Integration Testing | End-to-End Testing |
|---|---|---|
Main question | Do these components exchange and process information correctly? | Can the intended user or caller complete this workflow? |
Typical evidence | Correct request mapping, persisted state, or consumed event | Correct outcome after the connected sequence of actions |
Dependencies | Real components at the examined boundary; other dependencies declared | Assembled path for the journey; external replacements and exclusions declared |
Failure interpretation | Smaller investigation area if assertions identify the failed handoff | Visible workflow consequence, potentially involving several handoffs |
Main maintenance work | Contracts, schemas, fixtures, and component configuration | Journey behavior, shared environment, data, and UI where included |
The invocation tool does not determine the type. A browser displaying a mocked registration response cannot establish that registration saved an account. Conversely, a sequence of API calls can cover a backend business journey. Call that API-level E2E if useful, but do not claim it covers the frontend's controls, rendering, or navigation.
A check against a test double establishes behavior under assumptions about a collaborator. Contract verification can check communication expectations against a provider's selected behavior. Neither, by itself, proves that the deployed components can communicate using their actual routing, credentials, queue, and database configuration.
Focused checks often require less setup, but an integration suite that starts expensive infrastructure can take longer than a short journey against a ready environment. Compare provisioning, execution, retries, and investigation rather than assigning guaranteed speed to a label. For broader coverage placement, see the test pyramid explained by tested behavior.
Consider this fictional application contract, not a customer case or a report of executed tests. A new person submits a unique synthetic email address and valid registration details. The application persists one pending account and emits a verification event for it. A worker consumes the event, and a mail adapter produces a message containing that account's verification link.
Before verification, correct credentials must not create an authenticated session. A valid, unexpired, single-use token activates its own pending account. An invalid or expired token leaves the account pending. After successful verification, a fresh sign-in must reach an authenticated view for the same identity. Token lifetime, resend behavior, and other account policies need separate requirements; this example does not prescribe them.
For safe nonproduction testing, assume the mail adapter sends to a non-delivering capture service. The tests can inspect the resulting message there. This boundary includes message generation and submission to the capture service, but excludes the production provider's delivery, filtering, and inbox placement.
Check | Components Exercised Together | Controlled or Excluded Dependencies |
|---|---|---|
Registration seam | Request handler, registration service, test database | Verification dispatch controlled; no browser required |
Message seam | Event producer, test broker, worker, mail adapter, capture service | Registration request and database setup supplied by fixtures; production mail provider excluded |
Activation seam | Verification handler, account database, authentication policy | Pending account and token prepared directly; message delivery excluded |
Browser onboarding journey | Browser, deployed registration and activation paths, database, broker, worker, mail adapter | Capture service replaces production delivery; all owned journey steps remain connected |
The replacements change what the result means. If the journey instead intercepts registration and returns a prepared success response, it cannot support the same persistence and activation claim. Document that reduced boundary rather than continuing to describe it as the complete onboarding journey.
Choose seams where information changes representation or ownership: request to record, event to message, token to account state. These checks can investigate variations without repeating the entire browser journey for every input.
Send registration through the actual request handler and service against an isolated test database. Apply the schema and migrations needed by that build. Assert the expected response, then inspect the stored account independently: the submitted email belongs to the returned account identifier, exactly one account exists for this attempt, and its state is pending.
Also attempt authentication through the real authentication path with the submitted credentials. Under the example contract, it must not establish a session yet. A response that says “pending” while the authentication service admits the account violates the intended behavior.
A mocked repository cannot demonstrate actual persistence or database constraints. A substitute database engine may behave differently from the production engine. Use the appropriate real engine when query, transaction, or constraint behavior is the risk, and retain any remaining environment differences in the test record.
Exercise the actual event producer with a known pending account, using the real event serialization, test broker, worker, and mail adapter. Supply unrelated account setup through a fixture. Assert that the consumer receives the intended account identifier and that the captured message is addressed to that synthetic identity with a link for its verification token.
This check can expose an event-field mismatch, incorrect routing, or a token mapped to the wrong account. An assertion that the producer called a mocked broker would establish less: it would not show that the broker accepted the message or that the worker could consume it.
Wait for the specific captured message under an agreed time bound. Do not select whichever message arrived most recently in a shared inbox. Match the run and account so another test's output cannot satisfy the assertion. Production-provider behavior remains outside this seam; investigate it with separately authorized, explicitly scoped provider checks when that evidence is needed.
Prepare a pending account and token directly, then call the actual verification handler against the test database. Direct setup is appropriate here because the test is about activation, not token delivery. Check that a valid token activates its own account, leaves an unrelated pending account unchanged, and cannot be used again under the example's single-use rule.
Use independent fixtures for invalid and expired tokens. Assert the specified rejection and unchanged pending state, then verify that authentication remains blocked. These are negative cases at the same integration boundary, not a different testing type.
Keep pure token-decision permutations near their logic where possible. Retain integration cases that establish actual stored expiry handling, account selection, and state transitions. A passing mocked validator is not evidence that the handler updated the right database record.
The browser journey must connect the stages that the integration fixtures deliberately separate. Begin with a genuinely unregistered synthetic identity. Pre-creating an active user and loading a saved session would skip the behavior this journey is meant to establish.
Supplementary API or database observations can strengthen the browser assertions without replacing them. Playwright's API testing documentation, for example, describes checking server-side postconditions after browser actions. The tool's presence does not make an assertion sufficient: a success banner alone might appear even though no account was activated.
Keep a separate rejection journey when it adds user-facing evidence. An expired-link case could verify that the browser shows the agreed error and offers only the behavior specified by the product, while the account remains pending and sign-in remains blocked. Its setup can deliberately create an expired token because that case claims to test expired-link handling, not the full registration-to-delivery sequence.
Happy and unhappy paths describe scenario intent. Integration and E2E describe scope. Either scope can cover success or rejection. Select E2E rejection paths where the assembled behavior matters, rather than either banning them or replaying every invalid-token permutation through the browser.
Suppose an illustrative change renames the account-identifier field in verification events, but the worker still expects the old field. A message-seam check with actual serialization and consumption should reveal the incompatible handoff. An onboarding journey may instead stop because no matching message appears. Both failures are useful, but the seam's observations narrow the investigation.
Now suppose activation works when called directly, but the deployed verification link points to the wrong route. Focused activation checks may pass while the journey cannot follow the generated link. Depending on its assertions and configuration, the message-seam check might also detect the incorrect destination. Do not claim that only one test type can ever expose a particular defect.
A missing message is not automatically an application regression. Inspect the sequence:
In an environment using a provider sandbox, provider availability and configuration add further possibilities. Preserve the observed failure before naming its cause.
Record the build, environment, fixture version, account identifier, request or correlation identifier, relevant event, and redacted message evidence. Trace-context propagation can connect service observations where instrumentation supports it. Screenshots explain what the browser saw; correlated records explain how far the backend progressed. Protect credentials, session cookies, and verification tokens in retained artifacts.
Use bounded condition-based waits instead of arbitrary sleeps. Distinguish an assertion polling for eventual state from rerunning an entire failed test. Playwright classifies a test that fails initially and passes on retry as flaky. Keep the initial failure and investigate it; a later pass does not demonstrate a fix.
Choose the first check around the missing evidence. If onboarding has no credible deployed-outcome check and a controlled environment is ready, a critical journey can provide an initial signal. If a schema or event contract is changing, start with the seam that can establish that change precisely. Then add the complementary check whose absence leaves an important uncertainty. Neither integration-first nor E2E-first is a universal adoption rule.
Use change scope and prerequisites to decide execution timing. A registration mapping change calls for request-and-database checks; an event change calls for producer-consumer checks; a verification-route or session change calls for the affected journey. Shared authentication, migrations, dependency upgrades, and environment configuration can justify broader coverage even when individual test files did not change.
Run checks at the earliest stage where their required components and data are available. Selected journeys can run during pull-request validation when a suitable deployment exists; E2E is not inherently a release-only activity. Keep a broader regression policy because change-based selection can miss indirect dependencies. Playwright's CI documentation explicitly describes affected-test selection as a heuristic and recommends the full suite after the preliminary selected run.
Make both layers reproducible. Allocate synthetic identities and backend records per test or worker, isolate broker messages and captured mail, and verify readiness before execution. Browser cookies are only part of isolation. Reset the databases, messages, and other mutable state the test owns without erasing another worker's evidence. Preserve diagnostics before cleanup and define expiry-based cleanup for abandoned runs. The guide to safe, repeatable QA data develops dataset contracts, provisioning, and reset boundaries.
For Jira-based teams, Sherlock Test Management can support the reviewed test-definition side of this work. Its current product documentation describes proposed cases with steps, expected results, and source references, human-approved versioned baselines, and release-specific runs. A reviewer could make the pending-account precondition and captured-mail boundary explicit before approving an onboarding case. Keep automation execution and environment provisioning in separately selected tooling; this test-management workflow does not establish a browser runner or CI integration.
For the next onboarding change, write down the connection or journey at risk, what remains real, the observable result, and the evidence gap. Add a focused integration check when the handoff needs proof, and retain an E2E journey when completion across the assembled path needs proof. The goal is an explainable coverage decision, not a prescribed ratio of test names.
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A tester reviewing a Jira issue may need a fresh smoke run without opening GitLab and choosing pipeline parameters. A manually triggered Jira Cloud…
A regression-results worker receives HTTP 429 while reading Jira issues. Sleeping that worker may be necessary, but it will not help if other workers…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.