Flaky Test Quarantine: Owners, Expiry, and Nonblocking CI Results
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A code scan flags a possible SQL injection, but a scan of the running application reports nothing. Before closing the warning or blocking the release, ask what each scan examined. The code analyzer may have identified a dangerous data path that the runtime scan never reached. It may also have misunderstood a protection that needs review.
Static application security testing, or SAST, analyzes source code or a compiled representation without exercising the target application as a running service. Dynamic application security testing, or DAST, examines a running application through its exposed behavior. Their findings answer different questions. Using both gives a team different kinds of evidence, but neither a finding nor a clean report establishes the security of the whole application.
The useful SAST vs DAST comparison starts with the material each method can inspect. A static analyzer reasons about a representation of the program. A dynamic test observes the application under particular requests, identities and environmental conditions.
Comparison Criterion | SAST | DAST |
|---|---|---|
Input Inspected | Source or compiled code, selected rules and supported models | A running target, reachable requests, test data and relevant application state |
Main Prerequisite | Access to analyzable code and support for its language and frameworks; a build may be required | Authorized access to a running environment and the ability to reach the behavior being tested |
Possible Injection Finding | A modeled flow from untrusted input to database query construction | Response behavior suggesting that an input changed how a query was interpreted |
What the Finding Establishes | The analyzer identified a code pattern or path that warrants investigation | The tested application produced the recorded behavior under the tested conditions |
Important Blind Spot | Missing or inaccurate models, excluded code and deployment context outside the analyzed material | Unreached endpoints, inputs, roles or states, and behavior outside the enabled tests |
These boundaries overlap. Both methods can contribute evidence about the same SQL injection. However, a code location does not establish that a vulnerable route is deployed, and a response from one route does not establish that every related code path was examined. Compare scope and evidence before comparing alert totals.
Consider a fictional support application. An agent enters a customer name into a search form. A request handler passes the value to a database helper, which builds a query by concatenating the name into SQL text. The intended behavior is to find matching customer records within the agent's permitted scope.
The injection risk comes from mixing input data with query structure. If the database interprets part of the supplied name as SQL rather than as a value, the input can change the query's meaning. The interface might look ordinary while the unsafe construction sits several helper calls away from the form.
This is an illustrative vulnerability, not a report of a tested application. Assume the search exists, its query is assembled unsafely, and the team is evaluating how each method could reveal that flaw. No scanner output or successful exploitation is implied.
The intended repair is to keep the query structure separate from the customer-name value through proper parameter binding. Input validation can supplement that defense, but checking that a name looks reasonable does not replace parameterization. Similarly, calling a database helper, an ORM or a stored procedure does not establish safety if its implementation still assembles unsafe dynamic SQL.
SAST can investigate the customer-search implementation before a test environment is available. Its advantage is access to the code that constructs the query. The quality of the result depends on which code was analyzed and how accurately the analyzer represents the relevant behavior.
The team needs the relevant source revision or supported compiled artifact, appropriate rules, and support for the application's language and framework. Some analysis workflows also require a working build and its dependencies. Confirm those requirements for the chosen tool rather than assuming every SAST product operates the same way.
For this search, a data-flow analysis could follow the customer-name value from the request handler through the helper to the database operation. The entry point for untrusted data is commonly called a source. An operation where that data could cause a security problem is a sink. Taint tracking follows the influence of untrusted data even when intermediate operations change its value.
A useful finding would identify the input, the query-building location and the relevant path between them. That is a modeled path, not a recorded interaction with a deployed database. It gives the developer a place to investigate without requiring the runtime scanner to discover the search form first.
Analysis scope also matters. CodeQL's documentation, for example, distinguishes local data flow within a function from global flow across functions and object properties. This does not mean every SAST engine behaves like CodeQL. It illustrates why a label such as "data-flow analysis" is insufficient evidence that a particular helper boundary was covered.
Review the reported path against the actual implementation. Does the request value reach the database operation? Is it concatenated into query structure or bound separately? Has the analyzer represented the helper correctly? OWASP's source-code analysis guidance notes both false positives and the difficulty of proving that an identified issue is an actual vulnerability.
A safe parameterized helper that the analyzer misunderstands could explain a false positive. A genuinely unsafe helper behind a disabled feature is a different case: the deployment may reduce current exposure without making the implementation safe. Record that distinction instead of suppressing both findings under the same explanation.
SAST can also miss the flaw. Excluded files, unsupported behavior or gaps in library modeling can interrupt the analysis. CodeQL documents challenges involving unavailable library source, runtime-dependent behavior, aliasing and the size of data-flow graphs. A completed scan therefore needs an account of its analysis scope, not an assumption that every possible path was checked.
DAST approaches the fictional search through the running service. It can provide evidence about how the assembled application responds, but only if the test reaches the relevant action and supplies conditions that expose the unsafe behavior.
For an authorized test of this example, the team would need a running deployment, safe test records, the customer-search endpoint and a permitted test identity that can use it. If the search requires a session or prior workflow state, reaching that state is part of the test's prerequisites. Receiving a login page is not evidence that the customer-search query was exercised.
An active dynamic test could vary the search input and inspect the resulting responses. A meaningful alert should retain the request, the observed response and the reasoning connecting that behavior to suspected injection. Depending on the test, a response difference might support further investigation without demonstrating that sensitive records were retrieved.
This is observed behavior under specific conditions. It can strengthen the case that the flaw is exposed in that deployment, but the reviewer still needs to assess what the observation supports. An error response alone should not be expanded into a claim of complete database compromise. Nor does the alert necessarily identify the precise helper that needs repair.
Suppose the dynamic scan visits the application but never submits the customer search. Its report says little about this query. The same problem arises if the test identity lacks access, the session expires before the action, the relevant input is excluded, or the enabled tests do not expose the flaw. These are coverage questions to investigate, not evidence that this particular scanner suffered those failures.
Record whether the intended action was reached and what request was tested. Also distinguish the test deployment from production. A different source revision, feature configuration or database account can change the conditions under which the same unsafe query is exposed and the damage it could cause.
Active scanning carries operational risk. ZAP's Active Scan guide describes using known attacks against selected targets and warns that active scanning is an attack on those targets. Any such work needs explicit authorization and a controlled scope. A team should agree on safe data, permitted actions and stop conditions before testing, rather than treating the scan as a harmless read-only check.
Different results are useful when they direct the next investigation. For the same fictional customer-search flaw, consider these possible observations:
These are alternative hypothetical outcomes, not measured detection rates. They show why finding counts cannot determine which method is "better" without reference to the application and the question being investigated.
Business intent still needs contextual review. In the search example, deciding which customer records an agent should see requires knowledge of the permission model. Correct parameterization addresses the injection mechanism; it does not by itself prove the search enforces the intended access boundaries. Automated checks can test defined expectations, but a generic scanner result cannot establish that the team's business rules are complete or correct.
Once the team understands the customer-search flaw, use both perspectives to evaluate the repair. Keep the verification tied to the same behavior instead of replacing one unexplained alert with a new clean report.
The appropriate release policy depends on the system's risk and evidence, not a universal rule that one scanner must run first or every alert must block delivery. Use SAST to investigate unsafe implementation paths and DAST to investigate exposed behavior. For the customer search, the defensible conclusion is a scoped statement about the reviewed code, tested deployment and remaining gaps, never "the scanners passed, so the application is secure."
A confirmed flaky test can block unrelated changes without providing a dependable release signal. Removing that obstruction is reasonable only if someone still sees…
A tester reviewing a Jira issue may need a fresh smoke run without opening GitLab and choosing pipeline parameters. A manually triggered Jira Cloud…
A regression-results worker receives HTTP 429 while reading Jira issues. Sleeping that worker may be necessary, but it will not help if other workers…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.