Flaky Test Quarantine: Owners, Expiry, and Nonblocking CI Results
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
Your checkout change is ready to test, but the stock-reservation provider's sandbox is unavailable. Waiting blocks feedback on your own code. Replacing every dependency with a response that always says “accepted” creates a different problem: the tests can pass while the application sends the wrong request or mishandles a timeout.
Service virtualization gives the application a controlled substitute for that dependency. You can exercise selected success and failure conditions without waiting for the provider, provided you define what the substitute represents and preserve the checks that still require the real service.
A virtual service simulates the externally observable behavior of a dependency. For an HTTP API, it receives requests and returns configured responses over HTTP. Your application uses its HTTP client and response-handling code, but its configured endpoint points to the simulation instead of the unavailable upstream service.
This does not require reproducing the provider's entire implementation or virtualizing its operating system. Model the operations and conditions needed for the consumer behavior being tested. If checkout needs reservation creation and lookup, implementing unrelated supplier-administration endpoints adds maintenance without improving those checks.
The labels overlap. A stub supplies a configured response; interaction verification checks how a dependency was called; a virtual service provides a controllable dependency boundary. An HTTP mock server can combine these capabilities. WireMock, for example, describes its request-response mappings as stubs while also supporting service virtualization, request verification, stateful scenarios, and fault simulation. Do not select a tool on the assumption that anything called a mock must run inside the application or that a stub cannot match request details.
A simulation pass is evidence about the consumer under modeled conditions. It does not establish that the real provider accepts the request, returns the modeled response, or behaves correctly in the deployed environment.
Consider a fictional checkout application that asks a stock service to reserve an item. For this example, the agreed API returns a reservation identifier when stock is available and a conflict response when it is not. Checkout must preserve a pending outcome when confirmation is uncertain, rather than inventing a successful reservation. These are illustrative requirements, not universal inventory-API rules.
Before authoring the virtual service, collect the relevant API specification, provider examples, and decisions agreed with its owner. Record which version supports each behavior. Where the provider has not decided an error response or retry rule, identify an assumption for review instead of turning a convenient fixture into an undocumented contract.
Define the minimum useful model:
Choose matching rules carefully. A mapping that accepts any request body can hide a missing quantity or an incorrect item identifier. Matching an entire serialized JSON string can make harmless formatting differences fail. WireMock supports semantic JSON matching and more selective predicates, including matching values within JSON. Decide which differences are meaningful to the contract rather than making every matcher permissive.
An OpenAPI document or JSON schema helps describe structure. It cannot, by itself, establish that a reservation belongs to the requested warehouse or that retrying a request avoids duplicate stock allocation. Those relationships need explicit behavior and assertions.
Use a dedicated test configuration that can change the dependency endpoint without changing production configuration. The application, virtual service, and test driver must be able to reach one another. In a containerized setup, an address that refers to the application's own container will not automatically reach a separate virtual-service container.
A practical setup sequence is:
WireMock's default response for an unmapped request is an HTML response with a 404 status. Treat that as a possible setup or matching failure, not automatically as the provider's documented “item not found” response. If you add a catch-all mapping, its requests may count as matched, so checking only for unmatched traffic is insufficient. Verify the intended mapping and meaningful request fields too.
Overlapping mappings also need review. A general success mapping must not override a specific failure case. Use explicit priorities where appropriate and inspect which rule served the request. A healthy virtual-service process proves that the process started; it does not prove that the correct scenarios were loaded.
Each scenario needs a chosen dependency condition and an independently specified consumer outcome. Changing responses until checkout turns green can encode the same mistake in both the simulation and the test. Keep the business requirement as the basis for the assertion, and use the virtual service to supply the condition.
The following matrix continues the fictional checkout example. Its status choices and expected actions are example contract decisions to agree with the provider and product owner before implementation.
Dependency Condition | Virtual-Service Behavior | Consumer Observation to Check | What Remains Unproved |
|---|---|---|---|
Reservation accepted | Return the agreed 201 response and a stable reservation identifier | Checkout records that identifier and shows the confirmed reservation | The real provider accepts the request and persists the reservation |
Insufficient stock | Return the agreed 409 response and business-error body | Checkout offers the defined alternative without confirming stock | Actual provider inventory and stock-contention behavior |
Response arrives too late | Delay a valid response beyond the client's configured response deadline | Checkout follows its timeout policy and preserves an uncertain outcome | Real network latency and whether the provider completed the operation |
Temporary service failure | Return the agreed 503 response | Checkout follows the approved retry or fallback policy without unbounded calls | Real provider recovery and production retry interactions |
Invalid response | Return a malformed payload or the selected transport fault | Checkout handles the parse or transport failure without inventing success | Every possible corruption or client-platform behavior |
Repeated reservation request | Model the agreed idempotency identity and state transition | The repeated operation has the required consumer-visible outcome | Real-provider deduplication and concurrent allocation correctness |
The last case requires more than matching the presence of an idempotency header. The model must connect repeated requests to the relevant state and response. If it cannot represent that relationship, keep the case outside its supported scope and test the remaining consumer behavior separately.
WireMock's request journal can verify received requests and request counts. Use that evidence to inspect whether retries occurred and whether they used the intended identity. Also assert the application's final state or visible result: the expected number of HTTP calls alone does not prove that checkout handled the response correctly.
A delayed response, an HTTP error, and a broken connection exercise different paths. WireMock documents fixed and random response delays, chunked delivery, empty responses, malformed response chunks, and connection-reset faults. Select the mechanism that reaches the client behavior you intend to investigate.
A fixed response delay does not reproduce a DNS failure, a TLS handshake failure, or failure to establish a connection. Likewise, an HTTP 503 is a valid HTTP response, while an empty or corrupted response may fail before the client obtains a usable status and body. Keep separate assertions for these conditions where the application distinguishes them.
Check platform limitations before relying on a specific transport fault. WireMock documents that its connection-reset-by-peer fault works properly on Unix-like systems, while Windows may hang rather than reset. Confirm the observed client error in the intended test environment instead of assuming the configured fault name proves the failure occurred.
Start with controlled, repeatable profiles around the application's configured deadlines. Randomized delays can support a separate exploration, but preserve the generated conditions needed to investigate a failure. Configuring a delay does not measure the real provider's latency or demonstrate production throughput, capacity, or recovery under load.
Stateful behavior is useful when a request changes what the next request should see. WireMock scenarios provide a state machine in which mappings can depend on the current scenario state and advance it. A reservation flow can therefore distinguish an initial request from a later lookup, within the behavior you explicitly model.
Do not assume that naming a scenario creates isolated state for each account, reservation, or parallel worker. A shared scenario can let one test change another test's starting conditions. Use an isolated virtual-service instance per test scope where practical, or design and verify a state-partitioning scheme that the selected tool supports. A global reset in a shared instance can disrupt unrelated tests.
Keep data relationships consistent across the application and simulation. The item and warehouse in the request should correspond to the response data; a returned reservation identifier should be usable in the modeled lookup. Prefer deliberate synthetic fixtures for these cases. If recording is permitted, review request and response bodies, headers, cookies, query strings, body files, and logs for sensitive information before retaining or sharing them.
Recording is a starting point, not a complete behavior catalog. It captures interactions that occurred while the real service was accessible. Rare errors still need deliberate modeling. Review generated matchers too: WireMock's recording defaults for JSON bodies ignore array order and extra elements, which may be inappropriate for a particular contract.
Separate reset from refresh. Reset returns a scenario to its known starting state; refresh changes the approved dataset or behavior definition. Record the simulation revision, dataset version, initial state, and application build with each result. Keep request evidence before reset, because WireMock's journal is in memory and can be cleared or disabled. For fixture relationships, isolation, and reset procedures, see the practical guide to safe, repeatable QA data.
A simulation can become consistently wrong. If the provider changes a required field while the virtual service continues accepting the old request, the consumer suite may remain green. Version the mappings with their contract evidence, assign an owner, and review them when provider specifications, consumer requirements, or observed provider behavior change.
Keep different verification results separate. Consumer tests against a virtual service establish behavior under its model. Provider contract verification checks whether the actual provider implementation satisfies represented interactions. Selected real-service integration checks investigate the configured connection, identities, permissions, and cross-system behavior that the simulation bypasses.
Pact illustrates the contract-verification distinction. Its consumer tests exercise request creation and response handling against a mock and produce a contract artifact. Provider verification then replays the represented interactions against the provider implementation. Passing the consumer test alone does not complete that second step. Pact's documentation also limits this scope to the communication contract rather than UI behavior or general business logic; provider-side downstream dependencies may still be mocked.
Retain targeted checks against the real dependency when an approved environment becomes available, including the business paths and deployment-specific risks that matter for the change. A schema comparison is useful but cannot settle stock semantics, authentication behavior, concurrent updates, or an operation's real side effects. Investigate discrepancies rather than automatically updating the virtual service to match whichever result is easier to obtain.
Record the last provider comparison, relevant versions, known simulation gaps, and outstanding real-service checks. If the provider remains unavailable, report the consumer scenarios as tested against the named simulation and the required real-dependency checks as blocked or pending. Do not relabel that combination as validated real integration.
Choose one unavailable operation, agree its contract slice, and make success, rejection, timeout, and failure conditions deliberate. Establish reliable routing and reset before expanding the model. The useful result is a consumer behavior you can inspect and repeat, with a named owner and a clear record of what still needs the real provider. That lets testing proceed without allowing the substitute to become the only source of truth.
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A tester reviewing a Jira issue may need a fresh smoke run without opening GitLab and choosing pipeline parameters. A manually triggered Jira Cloud…
A regression-results worker receives HTTP 429 while reading Jira issues. Sleeping that worker may be necessary, but it will not help if other workers…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.