A release summary says testing is almost complete and the remaining failures are low risk. The sentences sound reasonable, but one passing result belongs to yesterday's build, a blocked refund check has disappeared from the totals, and the supposed risk acceptance exists only in a comment asking for approval. Sending that summary would give the release meeting a misleading starting point.
AI can help turn a verified evidence packet into readable prose. It should not decide which records qualify, calculate the authoritative totals, or convert missing information into reassurance. OpenAI describes hallucinations as plausible but false statements generated by language models. For QA reporting, the practical response is to make every consequential statement traceable and leave unsupported conclusions explicitly unresolved.
The workflow below separates source collection, deterministic calculation, narrative drafting, and human review. It applies whether the final summary is a short stakeholder update or the narrative section of a larger test report.
Start with the question the summary must answer. “What evidence do we have for this candidate, and what still prevents the agreed testing conditions from being met?” is more useful than “Summarize release quality.” The narrower question establishes what belongs in the report and what requires separate judgment.
Write a report contract containing the release identifier, candidate build or source revision, environment, relevant configuration, test baseline revision, selected run identifiers, and evidence cutoff with a time zone. State the counting unit. A unique case, a case-environment pair, and an execution attempt produce different totals even when they describe the same testing effort.
Also retain the selected scope and exclusions. An intentionally excluded browser is different from a required browser whose tests never ran. Record the reason and actual approval reference for an exclusion; a request to remove difficult work is not an approved scope change. If the scope changes after the cutoff, create an identifiable revision rather than quietly changing the denominator beneath an existing report.
Keep platform indicators separate from test outcomes. Jira Cloud's release page summarizes work-item and related-work status. Its development warnings depend on connected tools, appropriate permission, and enabled warnings. A work item marked done does not establish that all required tests passed, and the absence of a displayed development warning does not establish readiness. Test-management apps and external runners hold execution evidence that needs its own interpretation.
A source packet should let a reviewer reconstruct the summary without trusting the model's memory. Give each retained record a local evidence ID and keep the reference back to the authoritative source. An ID helps locate evidence; it does not prove that the evidence supports a claim.
For a test outcome, retain the case and run identity, executed case revision where available, candidate, environment, recorded status, execution time, and supporting evidence reference. For a defect, retain its identity, current treatment, affected behavior, relevant candidate, and owner. For a waiver or risk acceptance, retain the actual decision, authorized decision-maker, scope, and timestamp. Do not infer acceptance from a proposed workaround or a discussion thread.
Record the extraction time separately from the time a test ran. A freshly downloaded result can still describe an older candidate. Conversely, a recent defect change may not yet appear in the search result used to assemble the report. Atlassian documents eventual consistency for enhanced JQL search, so record the collection method and recheck consequential recently changed records through an appropriate source read rather than assuming every view updates simultaneously.
Check that the collector reached the end of the applicable result set, preserved filters, and accounted for duplicate records. If several exports overlap, deduplicate by the agreed reporting unit before calculating anything. A partial response, unavailable run, or inaccessible document is a completeness limitation. It must not become “no failures found.” Ask the source owner to reconcile the gap or disclose the narrower evidence boundary.
This does not require giving an AI assistant broad access. A controlled export containing the necessary fields may be safer and easier to review than unrestricted retrieval. Record which inputs were unavailable and ensure the intended reviewer can inspect the supporting records. Do not copy restricted defect details into a wider stakeholder summary merely because the report author can access them.
If the team uses Sherlock Test Management, its current product description identifies useful inputs: versioned test baselines, release-specific runs, recorded Pass/Fail/Blocked outcomes, linked Jira defects, and release reports. Use the actual executed baseline and run references when assembling the packet. The Sherlock product description explains those connected records.
That app workflow is separate from native Jira indicators and from any externally configured narrative-writing assistant. These capabilities do not establish an automatic AI release-summary feature, autonomous browser execution, or a supported connection to an arbitrary runner. Verify the collection method in the installed setup before designing a reporting workflow around it.
Use a spreadsheet formula, query, or small deterministic program to normalize statuses and calculate totals from the retained records. Define how the source statuses map into report categories. Keep unsupported or inconclusive states visible instead of forcing them into Pass. Have a person review the mapping before using it repeatedly.
Choose the qualifying-result rule before counting. For example, select the latest qualifying outcome for each case-environment pair on the assessed candidate before the cutoff, while preserving earlier attempts. A later pass on a different build cannot silently replace an unresolved result on the candidate being assessed. Preserve links between retries so a passing final outcome does not erase a material earlier failure or a suspected flaky check.
Fictional worked example. The following release, records, and findings are illustrative, not results from a real Jira project, test run, or AI experiment. Candidate CANDIDATE-11 uses baseline BASE-11-r2 in staging-refund. At a fictional 14:00 UTC cutoff, the selected scope contains 50 case-environment pairs, each classified once by its latest qualifying outcome.
Latest Qualifying Outcome | Selected Units | Interpretation |
|---|---|---|
Pass | 40 | Qualifying passing evidence exists |
Fail | 4 | Qualifying failing evidence exists |
Blocked | 3 | Required checks remain unresolved |
Not Run | 3 | No qualifying execution was recorded |
Total | 50 | Every selected unit remains visible |
Under this example's convention, completed verdicts mean Pass plus Fail. There are 44 completed verdicts, and the pass rate among them is 40 divided by 44, or 90.9% rounded to one decimal place. The passed share of selected scope is 40 divided by 50, or 80%. Completed verdicts cover 88% of selected scope. These are different measures, not interchangeable descriptions of completion.
All three blocked units in this fictional packet have an attempted execution. Five additional retries bring the total to 52 attempts, but the selected denominator remains 50. The four failing units link to three distinct unresolved defects because two failures share one defect. Count defects by defect identity, not by failed-test rows.
Suppose the most consequential defect permits a duplicate refund after a timeout. A high pass percentage does not make that impact acceptable. Nor does it resolve the blocked dependency or the unrun checks. If a passing record from an older candidate appears in the input, flag it as nonqualifying and retain the reason; do not use it to improve the current candidate's rate.
When there are no completed verdicts, report the pass rate as not applicable. Do not invent zero percent or a successful outcome. Keep the agreed definition beside the result, particularly when an app uses “executed” differently from this example's completed-verdict convention.
Once the packet reconciles, supply the verified metrics, evidence index, scope contract, and reviewed risk records to an approved writing tool. Ask for explanation and organization, not new calculations or independent release authorization. Keep the calculation output authoritative if the generated prose disagrees with it.
A reusable instruction can be written in plain language:
Draft a QA status summary for the audience named in the report contract. Use only the supplied scope, computed metrics, evidence records, and recorded decisions. Preserve the candidate, environment, baseline, and cutoff. Associate each material factual statement with the relevant evidence IDs in the review copy. Copy supplied counts and rates without recalculating them. Separate observed outcomes, unresolved evidence, and human recommendations. Do not infer a pass, risk acceptance, owner commitment, or release approval. If a required fact is missing or contradictory, identify the gap and ask for review. Do not publish, send messages, or modify source records.
This is a proposed instruction, not evidence that a particular model follows it reliably. Retain the prompt version and generated draft so reviewers can see what changed. A short executive summary can omit technical detail for readability, but it must preserve material blockers and point to a fuller evidence record.
Treat issue descriptions, comments, logs, and retrieved documents as untrusted data. A comment telling the assistant to ignore failures or send the report elsewhere is content to assess, not an instruction from the report owner. Anthropic's guardrails guidance identifies this indirect prompt-injection risk and recommends clear trust boundaries and least-privilege access. Prompt wording alone is not an authorization boundary: the writing component should not have unnecessary sending, release, or source-editing permissions.
Before submitting inputs, check organizational rules for the approved provider and account. Remove secrets, personal information, and unnecessary confidential details. Preserve controlled references so authorized reviewers can still inspect the originals. Redaction must not leave the model inventing a substitute explanation; mark omitted context as unavailable where it affects interpretation.
Review the draft against the packet, not against whether it sounds like a plausible QA update. The NIST-hosted perspective paper on machine-generated reports distinguishes completeness, accuracy, and verifiability. Those dimensions provide a useful review lens, although the paper is not a QA-specific effectiveness study of this workflow.
Use a repeatable review pass:
In the fictional example, “Testing is 90.9% complete and the release is approved” would fail multiple checks. The percentage describes pass rate among completed verdicts, not completion of selected scope, and no release approval was supplied. A defensible replacement is: “At the cutoff, 44 of 50 selected units have completed Pass/Fail verdicts. Forty passed, four failed, three are blocked, and three were not run. Three unresolved defects remain, including a duplicate-refund defect. Release decision pending.” The review copy should associate these statements with the computed totals, defect records, and decision record.
Reviewing only the citations that appear in the generated text will miss omitted risks. Compare the draft with a required-content list derived from the report contract, including exclusions and missing evidence. For a fuller record, retain results alongside open risks and exit criteria; the Jira test summary report template provides a useful structure for that packet.
The reviewer should reopen consequential evidence, challenge unsupported reassurance, and confirm that the summary matches its stated boundary. A second model can suggest possible contradictions, but its agreement is not independent confirmation of the source records. Someone accountable for the report must resolve material discrepancies or disclose them before circulation.
Sharing gate. Record who reviewed the summary, which revision and evidence cutoff they reviewed, and any unresolved limitations. QA review confirms the report's evidence and interpretation within that scope. Business risk acceptance and release authorization belong to the people with that authority; neither a polished narrative nor a completed review field creates their decision.
Retain the reviewed summary with its source packet, calculation output, and corrections. If a new candidate arrives, a critical defect changes state, the selected scope changes, or a required result becomes available, assess whether the report needs an amendment. Give the amendment a new revision and cutoff, and explain the changed conclusion rather than silently replacing the earlier record.
Before the next status meeting, choose one consequential claim and follow it back through its evidence ID, qualifying result, candidate, and recorded decision. Then check one required risk that the draft did not mention. If either check fails, repair the summary or state the limitation before sharing it.
Two reports can describe the same symptom while disagreeing about where the failure occurs. If a reviewer accepts a duplicate suggestion because the titles…
A checkout test times out while waiting for an order confirmation. The runner reports an assertion failure, the application records a failed request, and…
A recipient-preview rule changes, but the regression suite still checks the earlier behavior. An AI agent may choose a different route through the editor…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.
