A refund policy changes before a release. Your team needs to update test coverage, decide what to run first, investigate failures, and explain the remaining risk. An AI assistant can produce a plausible plan for each task. You still need to determine whether it read the correct policy, preserved the permission checks, and distinguished an untested scenario from a passing result.
For a test manager, AI in software testing means applying AI techniques to testing work: drafting test ideas, analyzing existing evidence, recommending priorities, or assisting execution through specifically equipped tools. The useful unit of evaluation is the accepted work product. A long list of generated cases has little value if reviewing it takes longer than designing the relevant checks yourself.
Start with a task whose inputs, output, reviewer, and consequences you can define. Planning, prioritization, defect triage, and reporting offer practical starting points, provided their review gates remain explicit.
Different AI systems solve different problems. A predictive model can learn patterns from historical changes and test outcomes. A large language model can draft cases or summarize supplied evidence. An agent can call tools and take actions within the access it receives. A product that drafts test cases does not thereby demonstrate that it can operate your application or execute your CI suite.
Testing an AI system is a separate activity. Evaluating a chatbot's unsafe responses or a model's behavior across different populations concerns the system under test. Using an assistant to prepare a regression plan concerns the testing process. Both require evaluation, but they have different objectives and evidence.
Define the output narrowly: a draft coverage map, a proposed execution order, a set of duplicate-bug candidates, or a release-summary draft. Attach the source snapshot and review decision to that artifact. This gives a reviewer something concrete to accept, amend, or reject rather than an open-ended claim that the tool improves quality.
Fluent explanations need verification. NIST's July 2024 Generative AI Profile describes confabulation, where generated content is confidently wrong, and automation bias, where people defer excessively to automated output. A source citation helps a reviewer check a statement; its presence alone does not establish that the cited text supports the statement.
Use the same discipline across these tasks: supply relevant evidence, ask for a constrained proposal, and place a named person at the decision point. The refund examples below are illustrative scenarios, not reported customer results.
Provide the approved acceptance criteria, relevant product rules, existing cases, and known exclusions. Include source identifiers and versions so the assistant can distinguish current requirements from earlier discussions. Ask for test conditions, references to supporting rules, and unresolved questions before requesting detailed steps.
Suppose the new policy requires finance-role authorization before any refund attempt, including a retry after a gateway timeout. A useful proposal separates authorization, retry behavior, and duplicate-refund prevention. It also asks what should happen when the gateway accepted a request but its response was lost, rather than silently inventing a retry rule.
The failure mode is a plausible expected result unsupported by the specification. An assistant may also restate the happy path in several cases while omitting the authorization boundary. A tester must compare conditions with the actual rules, remove duplicates, and have the product owner resolve ambiguity before approval. Ask whether a case would expose a violation of the rule, not whether its wording sounds complete.
The expected result needs an independent basis, sometimes called a test oracle: the specification, agreed business rule, or other trusted reference used to decide correctness. Deriving the expected outcome solely from current application behavior can preserve an existing defect. Generated test code also needs inspection and execution in the real project; successful generation is not an execution result.
Supply the release changes, requirements-to-test relationships, business impact, previous failures, test duration, dependencies, and environment constraints. Historical failure data should distinguish confirmed defects from infrastructure problems and unreliable tests. Otherwise, a ranking may favor noisy checks for reasons unrelated to product risk.
Ask for the proposed order and the evidence supporting each inclusion or exclusion. A language model's explanation of priority is a recommendation, not a calibrated probability that a test will fail. A newly introduced payment path may have no failure history precisely because it has never existed before.
For the refund release, authorization and duplicate-payment protection may deserve early execution even if their previous runs passed. Shared gateway behavior can justify checking a related flow that had no direct story change. The test manager should challenge those dependencies with developers, preserve mandatory checks, and record what remains outside the time budget.
An early pilot can reorder work while leaving the broader regression commitment intact. Removing tests from release scope has a different consequence and needs a separate approval. Keep the excluded cases and reasons visible so faster execution does not conceal an unexamined risk.
Useful inputs include the failing step, expected and observed behavior, build and environment, timestamps, relevant logs, reproduction steps, and related defects. Ask the assistant to extract evidence, identify possible duplicates, and propose hypotheses with references to the supporting observations.
Two refund failures can share a timeout message while having different causes. One may involve an unavailable gateway; another may involve incorrect retry handling. Conversely, different messages may be downstream symptoms of the same incident. Similar wording is a reason to investigate a possible relationship, not sufficient evidence to merge or close reports.
A developer should test the proposed cause against a reproduction, trace, or targeted investigation. A tester should verify the affected behavior and retest an implemented fix. Keep severity, which describes impact, separate from scheduling priority. The accountable team decides both using current business context, rather than accepting the assistant's labels as established facts.
Record rejected hypotheses too. They show what was investigated and prevent the next summary from turning an earlier suggestion into an accepted root cause.
Supply a defined release scope, a result snapshot, build and environment identifiers, known exclusions, linked defects, and the rules used to calculate metrics. Compute totals from the system of record before asking an assistant to describe them. Require every claimed blocker, trend, or completed action to point to evidence in that snapshot.
A summary can mistakenly convert a case marked Blocked into a failure, omit unexecuted work, or describe a historical pass as evidence for the latest build. Specify whether a percentage uses all planned cases or only executed outcomes, and show the underlying counts. If the scope changes, explain that change before comparing percentages across reports.
The test manager reviews the draft against the source results and unresolved risks. A high pass rate cannot establish readiness when a critical permission check remains unexecuted. Requirements coverage also differs from code coverage: linking a case to a story says nothing by itself about which code ran or whether the case's assertions were adequate.
In Jira Cloud, Rovo can assist with generating and summarizing work-item text. That platform capability should not be confused with an installed test-management app's run records or an external runner's execution evidence. Confirm the available functionality and permissions in the intended environment before designing a workflow around it.
Approval changes what the team trusts. Drafts should remain distinguishable from approved cases, and suggested edits should remain distinguishable from the versions used in completed runs. Otherwise, a later AI-assisted rewrite can make an old execution appear to validate instructions that the tester never followed.
For each approved change, retain the source reference, proposed difference, reviewer, and decision. For each run, retain the executed version and actual outcome. When requirements change, review the affected cases and decide whether a new execution is needed. Editing a case cannot refresh historical evidence.
Sherlock Test Management provides a relevant product example. Its current product description describes using selected Jira issues, Confluence pages, and files to prepare a reviewed testing brief and proposed cases. People accept or reject additions and edits before the approved baseline changes. It also describes release scope based on Jira Fix Version, with priorities and assignments reviewed by the team.
Applied to the refund scenario, that workflow gives the reviewer a place to check the permission rule before accepting a proposed retry case. The same product page describes testers following steps and recording Pass, Fail, or Blocked in Sherlock Runner or supported Jira panels. These are vendor-described capabilities, not an independently measured productivity outcome or evidence of autonomous browser/API execution or a particular CI integration.
Evidence preparation is part of adoption. Defect reports and logs may contain customer identifiers, financial details, credentials, or confidential product information. Use an approved service, minimize the supplied context, and check its contractual data-use, retention, access, and deletion arrangements. Replacing a name does not necessarily remove every identifying detail from a log.
External text introduces another risk. A ticket or attached file can contain instructions that attempt to redirect an assistant. Treat retrieved material as evidence to analyze, not permission to change the workflow. A prompt asking the model to ignore malicious instructions is useful guidance, but downstream permissions must enforce the boundary independently.
For an initial management-assistance pilot, explicitly prohibit these actions:
Give the assistant read-only access where reading is sufficient. If a later workflow needs writes, restrict them to the necessary operation and resource, log each action, and verify its effect. Human approval should authorize a specific proposed change, not grant indefinite permission for whatever the assistant decides next.
Choose one recurring task with a manageable review burden, such as drafting a coverage map for refund-policy changes. Define what acceptable output looks like before seeing the tool's answer. Include straightforward inputs, missing information, conflicting sources, and a changed rule so the pilot tests more than an ideal example.
Use separate measures for usefulness, correctness, and operating cost. The following scorecard is a proposed starting point, not a published benchmark or universal acceptance standard.
Dimension | What to Record | How It Informs the Decision |
|---|---|---|
Source fidelity | Unsupported statements and incorrect source references per reviewed artifact | Reject outputs that invent essential rules or execution evidence |
Risk coverage | Relevant conditions found and known critical conditions missed against an independently reviewed reference | Check whether useful breadth survives beyond the happy path |
Review burden | Preparation, review, correction, and escalation minutes, including rejected outputs | Compare accepted work with the existing process |
Triage usefulness | Confirmed duplicate or cause suggestions divided by reviewed suggestions, plus missed known cases | Separate helpful leads from distracting noise |
Repeatability | Material differences across repeated runs using the same input and configuration | Identify unstable omissions and inconsistent decisions |
Operational control | Unauthorized actions, missing approvals, unavailable evidence, and stop-condition breaches | Decide whether the workflow can remain safely bounded |
Interpret these measures together. High agreement with source text does not prove that the source was complete. High precision among triage suggestions does not reveal how many important relationships the assistant missed. A reference set needs experienced review, and small pilot samples should not be presented as precise forecasts of future performance.
Consider a hypothetical effort comparison. The existing process takes 120 minutes. An assisted attempt needs 20 minutes of context preparation, 5 minutes of generation, 65 minutes of review, and 40 minutes of correction. Its total is 130 minutes, which is 10 minutes more than the baseline. Fast generation has not reduced total effort. The team might still value an additional valid risk discovered, but that benefit should be recorded separately rather than reported as time saved.
Run the pilot through a controlled sequence:
Set task-specific acceptance and stop conditions with the relevant owners. For example, an invented critical policy rule, fabricated execution result, or unauthorized write should stop promotion until investigated. A recurring review-time overrun may justify narrowing the task or abandoning it. Do not average a serious control failure into an otherwise favorable quality score.
A successful pilot supports only the task and conditions it evaluated. Good results on coverage drafting do not validate automated prioritization, root-cause diagnosis, or execution. Each expansion needs its own evidence and approval boundary.
Choose the next use case by whether your team can verify its output economically and preserve accountability. Drafting may be worthwhile even when autonomous action is inappropriate. If reviewers cannot reconstruct the source of a recommendation or inspect what changed, improve the evidence and controls before widening access.
For the next release, assign one bounded assistance task to a named reviewer and compare its accepted output, missed risks, and full effort against the current process. Expand when that comparison supports a concrete benefit and the agreed safeguards hold. Keep the release decision with the people responsible for its consequences.
Two reports can describe the same symptom while disagreeing about where the failure occurs. If a reviewer accepts a duplicate suggestion because the titles…
A checkout test times out while waiting for an order confirmation. The runner reports an assertion failure, the application records a failed request, and…
A recipient-preview rule changes, but the regression suite still checks the earlier behavior. An AI agent may choose a different route through the editor…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.
