Flaky Test Quarantine: Owners, Expiry, and Nonblocking CI Results
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A request form displays “Quantity must be 10 or less,” yet the request still appears in the approval queue. The interface rejected the input; the business operation went ahead. A test that checks only the error banner would miss the defect.
Negative testing examines how software responds to invalid inputs, unauthorized actions, and operations attempted in the wrong state. A negative test passes when the application handles the prohibited condition as required. A crash, an unexplained error, or an accidental data change is not a successful result merely because something failed.
For manual test cases, the useful starting point is the rule being protected. Identify what must be rejected, what must remain safe, and how the user should recover. The examples below use a fictional equipment-request portal. They describe proposed checks, not customer incidents or executed test results.
Before inventing invalid data, establish an agreed description of valid behavior. A quantity of zero is invalid for a purchase request but could be valid when removing an item from a basket. The field name alone cannot tell you the expected outcome.
For the illustrative portal, assume these requirements:
These assumptions define the examples, not a universal purchasing workflow. Your application may permit reopening, use different permissions, or process submissions asynchronously. Resolve those differences before copying a case.
Collect the relevant acceptance criteria, field limits, role rules, state transitions, and dependency behavior. Also identify how a tester can inspect the resulting business state. If nobody can explain whether a rejected submission reserves stock or creates an approval task, the expected result is incomplete. Ask the responsible product or engineering owner to define it rather than turning today's behavior into the requirement.
Use an authorized test environment, synthetic accounts, and isolated fixtures. Agree which actions, request-inspection tools, and dependency simulations are permitted. A manual negative-testing exercise does not authorize probing production accounts, sending destructive payloads, or interrupting shared services.
Several inputs can look different while exercising the same rule. Conversely, a perfectly formatted value can violate ownership or workflow policy. Organize candidates around the behavior they challenge so the suite does more than accumulate unusual strings.
For the integer quantity rule, values below 1 and above 10 challenge different limits. Choose 0 and 11 as adjacent invalid inputs, and retain positive controls at 1 and 10. Those controls help distinguish a correct boundary rejection from a form that rejects everything. This small selection is not a claim of complete boundary or input-space coverage.
Missing input, whitespace, text such as “many,” and a fractional value such as 1.5 may exercise different parsing or required-field rules. Specify each accepted representation. An empty string, an omitted property, and an explicit null are distinct conditions at a service interface, even if the browser presents only one empty field.
Use equivalence partitioning to group values expected to receive the same treatment, boundary value analysis for ordered limits, decision tables for interacting conditions, and state transition models for event order. The black-box test design techniques guide explains how those choices affect coverage. A representative value samples an assumed class; it does not establish that every member behaves identically.
Keep legitimate unusual data in view. An apostrophe in a person's name is not automatically invalid, and a validly formatted end date may still precede its start date. Derive rejection expectations from the field's purpose and business rules, not a blanket ban on punctuation.
Separate who is acting from which resource they are acting on. An employee attempting an approver-only action challenges role permissions. An employee opening a colleague's private request challenges ownership restrictions within the same role. Both need cases if both restrictions exist.
State adds another dimension. Approving a Draft skips submission; submitting a Cancelled request attempts a prohibited transition. Start each case from a verified state and normally attempt one invalid transition per case. An unexpected first transition can otherwise hide the reason a later action behaved incorrectly.
Use a compact inventory before expanding candidates into executable cases:
Condition Family | Illustrative Candidate | Protective Outcome to Specify |
|---|---|---|
Required input | Submit without a required justification | No submission; identify the missing field |
Numeric limit | Enter quantity 11 | Reject the quantity; preserve the Draft |
Role restriction | Requester attempts approval | No approval or approver-only data exposure |
Ownership restriction | Employee opens a colleague's private request | No private content returned |
Workflow state | Submit a Cancelled request | No new submission or approval work |
Duplicate action | Repeat the same logical submission | No duplicate business request or approval work |
This inventory identifies design targets. It does not demonstrate that any check has run or that every possible condition has been considered.
A disabled button or browser validation message tests the interface, not every enforcement layer. Browser checks can be bypassed. Where server enforcement is in scope, add a separately specified manual service-level check using an approved request tool and synthetic data. Keep its input, identity, expected response, and state verification explicit. Do not mark that check covered because the UI blocked an action first.
Replace “an error should appear” with a decision rule another tester can apply. Define the rejected operation, the required response, the protected business state, and the permitted next action. Safe rejection includes the non-effects that matter. A validation banner alongside an unwanted approval task is a failed case.
Consider case NT-QTY-01, “Reject an above-limit quantity without submitting the request.” Its requirement reference is the approved quantity-and-submission rule, and its purpose is to prevent an invalid Draft from entering approval.
Preconditions: an authorized employee is signed in; an isolated request is verified as Draft with quantity 3; every other required field contains valid data. The tester can inspect the saved request and its approval-work view through authorized access. No other process is editing this fixture. Record the starting state and existing work references before acting.
Follow these steps:
The correction step checks recovery after the invalid attempt. It belongs in this example because a stuck validation state could prevent a legitimate submission. Use a separate case when recovery branches become complex enough to obscure the original rejection check.
If the application processes work asynchronously, define the approved completion signal and observation limit before execution. An empty queue immediately after clicking does not prove that no task will appear later. Inspect the authoritative status or agreed diagnostic view, and use the documented observation window. If the necessary evidence cannot be obtained, record that limitation rather than claiming the non-effect passed.
Cleanup releases or resets the synthetic fixture through the approved QA procedure. Because the recovery step submits it, the next case needs a fresh Draft. Do not rely on the original fixture remaining reusable.
Expected results should also distinguish normal validation feedback from unexpected failures. A field error can explain how to correct the input; an unexpected exception should not expose stack traces, internal paths, or secrets. Permission-denial messages must follow the application's privacy policy and should not reveal protected record content merely to explain a refusal.
Reuse the structure of the quantity case, but change its oracle: the evidence that determines whether the outcome is correct. A private-record denial needs a content check. A cancelled workflow needs a state-and-side-effect check. A timed-out submission requires establishing what happened before deciding whether retrying is safe.
Prepare two synthetic employee accounts with the same role and a private request owned by one of them. Using the other account, attempt to open that request through an agreed test route. Expect the defined denial response and no private request fields in the returned content. Verify that the request remains unchanged.
Then open a request owned by the acting employee as a positive control. This helps distinguish an ownership restriction from a broken environment that denies all access. Keep credentials and session material out of screenshots and defect attachments. If request-level checks are outside the tester's authorization, record that exclusion; a UI-only result cannot establish enforcement at every endpoint.
Create a separate fixture, cancel it through the supported test workflow, and independently confirm its saved Cancelled state. Attempt submission using the approved entry point, including a previously opened form if stale forms are a relevant risk.
Under the illustrative contract, expect rejection, the request still Cancelled, and no new submission, approval work, or submission notification. A policy-approved audit entry may be created. Define these business non-effects explicitly instead of expecting every record in the system to remain byte-for-byte unchanged.
Recovery also follows policy. If cancelled requests cannot reopen, direct the user to create a new request rather than expecting a retry to revive the old one. Where incomplete operations reserve resources, agree how reservations are released or compensated. Do not assume an error can instantly reverse every external side effect.
A repeated click is not necessarily an invalid business request. The important requirement may be that repeated delivery of one logical submission produces one business outcome. Specify how the application identifies that logical action and what feedback a repeat receives. An error, the original confirmation, or an “already submitted” response may be acceptable under different contracts.
Test ordinary duplicate interaction in the approved environment, then inspect actual request and approval-work records. A disabled button alone is weak evidence. If a timeout leaves the outcome unknown, establish whether submission completed before retrying; a timeout does not prove that the operation failed. Idempotency is a design mechanism for controlling repeated effects, not something a tester should assume every application implements.
For stale data, prepare two sessions viewing the same Draft. Save a permitted change in one, then attempt an outdated update in the other. Agree beforehand whether the application must reject, merge, or otherwise resolve that conflict. Verify both the response and the final saved values. A manually arranged two-session check can expose lost updates, but it does not establish coverage of timing-sensitive races or high-concurrency behavior.
Keep the reusable case separate from its execution record. The case contains the approved rule, setup, actions, and expectations. Each run records the case revision, actual build and environment, fixture instance, acting role, observations, and outcome. Preserve earlier attempts when a fix is retested.
For a failure, capture the first discrepancy and its consequence. “Quantity error displayed, but request moved to Submitted and approval work was created” gives an engineer a more useful starting point than “negative test failed.” Attach redacted evidence showing the relevant response and saved-state comparison. Separate confirmed effects from checks that were blocked or not performed.
Prioritize cases where an incorrect outcome exposes private data, grants unauthorized power, corrupts records, or triggers damaging business activity. Add frequent user mistakes and known defect patterns, then examine lower-impact variations as risk and time justify them. There is no universal negative-to-positive test ratio or fixed number of cases that makes a feature safe.
Manual design is useful for interpreting ambiguous requirements, investigating workflows, and evaluating recovery. Stable validation and permission checks can later become automated regression tests. Broader input exploration, fuzzing, performance testing, and specialist security assessment address different coverage needs. Passing the selected manual cases does not establish exhaustive security assurance.
For the next important action you test, write down its rule, one focused prohibited condition, the business effects that must not occur, and the user's recovery path. If another tester cannot observe those outcomes, improve the specification or test access before adding more invalid inputs.
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A green automated test history can help a developer identify a safe-looking change while leaving a QA lead unable to answer a different question:…
A sprint can contain carefully estimated testing work and still overload the only person who can perform it. The missing input may be a…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.