A Jira item marked Done does not, by itself, tell a QA lead whether the checkout tests passed. The testing work may have finished with a known failure, stopped at a missing prerequisite, or left a critical scenario unattempted. A useful execution record separates those situations and identifies who needs to act next.
Jira test execution is the work of performing selected test cases against a defined build and environment, comparing actual behavior with expected behavior, and recording the results. In Jira-based testing, a test-management app supplies the run records and result model. Jira supplies the surrounding work tracking. Understanding which layer owns an assignment or status prevents a completed task from being mistaken for evidence of a successful test.
A test case describes what to check: prerequisites, actions, data, and expected results. A run records what happened when someone attempted that case in a particular context. The same checkout case can pass against one build and fail against another without changing its intended expectation.
Apps organize these records differently. In Xray Cloud, a Test Execution is a Jira issue type that collects tests. Each associated test has a Test Run, which is an internal Xray entity rather than another Jira issue. The execution can therefore have a Jira assignee and workflow status while the runs have their own testers and results. Do not assume another app uses the same object names or storage model.
Jira workflow statuses describe where a work item sits in a team's process. Done means the work on that item has finished according to that process. An app's execution result describes the test outcome. Unless your team has explicitly configured and verified rules connecting them, moving an execution issue to Done should not be interpreted as setting every run to Pass. Equally, recording a failed run does not establish which Jira workflow transition should happen next.
Before execution begins, define the context that makes the result interpretable:
Check what the installed app preserves. Xray Cloud copies selected test-specification and precondition fields into a run, but does not copy every Jira field. Important instructions left only in a test issue's description may therefore be missing from the run specification. Review the actual execution view rather than assuming the latest reusable case and an older result describe identical steps.
The execution owner coordinates the testing scope, timing, and follow-up. The run assignee performs a particular test. Those responsibilities can belong to the same person, but a shared release execution often needs different testers for payments, permissions, and recovery behavior.
Xray Cloud supports assigning the Test Execution issue and its individual Test Runs separately. Its documentation says runs are assigned to the execution issue's assignee by default, and assigning runs requires permission to assign that issue. Review the resulting run assignments instead of relying on the parent issue's name alone. Execution permission is another check: a person who can see or receive work may still lack permission to execute it.
A practical handoff gives the tester the intended build, environment, case version, data, and priority, along with a way to raise blockers. Assigning a reusable test case to its maintainer does not establish who should run it for the current release. Likewise, assigning the execution issue to a QA lead does not show that the lead personally performed every test.
Keep assigned to separate from executed by. One describes responsibility; the other records who performed the work. Xray Cloud exposes distinct run-assignee and last-executed-by fields. If work changes hands, preserve the observations and attempt history, and make the next responsibility explicit. Reassignment alone supplies no new evidence about the application.
The following meanings provide a useful vocabulary for discussing results. They are not a universal Jira status list. Your app may use different labels, allow custom states, or separate step results from an overall run result.
Common State | Meaning for This Attempt | Evidence to Retain | Next Action |
|---|---|---|---|
Pass | The tested behavior matched the expected result in the recorded context. | Relevant observations and the build, environment, and case version. | Continue with remaining scope and review unresolved risk. |
Fail | An observed result diverged from the expected behavior. | Expected versus actual behavior, failing step, and supporting evidence. | Triage the discrepancy and link the relevant defect when appropriate. |
Blocked | A prerequisite or impediment prevented a valid test of the intended behavior. | The unmet prerequisite, affected scope, and reason testing cannot proceed. | Assign an unblock owner and define when to retry. |
Not Run | No attempt has been recorded for this scoped test. | Its place in the planned scope and any documented scheduling decision. | Schedule it, investigate the omission, or explicitly revise scope. |
In Progress | Execution has started and the outcome remains incomplete. | Work completed so far and any outstanding steps or observations. | Finish the attempt or record the impediment that stopped it. |
For example, Xray Cloud documentation uses TO DO and EXECUTING alongside pass/fail outcomes. Its administrators can configure custom overall Test Run statuses, while custom statuses do not extend to test-step results in the same way. Map your team's reporting vocabulary to the installed configuration before combining results from different projects or apps.
Also check the aggregation rules. A step result, an overall test result, and an execution's progress summary answer different questions. Never assume the parent becomes failed after any failed step, or that a completed cycle means every test passed. A manually selected overall result may behave differently from one calculated from steps, depending on the app and settings.
The key question is whether the tester reached the behavior that the case was intended to validate. Similar symptoms can produce different records, so classification needs the test's expectation and the observed stopping point.
Suppose a payment test requires a working sandbox account, but the credentials have not been provisioned. The tester cannot reach the payment behavior. Record the missing access as a blocker rather than claiming the payment implementation failed.
Now suppose the sandbox is available, the tester submits an authorized payment, and the application displays an incorrect total. The intended behavior was reached and the result contradicts the expectation. That supports a failed result, with the expected total, observed total, relevant inputs, and evidence recorded together.
A blocker can still originate in a product defect. If a broken login prevents a downstream checkout test from reaching its assertions, the login failure and the blocked checkout run represent different observations. Link the dependency where supported. Marking every downstream case failed would imply that each behavior had been exercised and found wrong.
Not Run is less informative than Blocked. It says no attempt was recorded, without explaining whether the cause was access, scheduling, or a changed plan. If a tester tried to begin and discovered a missing prerequisite, document that discovery rather than leaving an unexplained untouched state. Where the app distinguishes Skipped or deliberately excluded work, keep that decision separate too.
A blocked label needs a route back to execution. The tester remains responsible for the test, while the person who can restore access, deploy the build, or supply data owns the unblock action. Naming only the tester can leave the dependency unattended.
Record the prerequisite, the evidence of its absence, the affected tests, the action owner, and the condition for retry. For example: the payment sandbox credentials are unavailable; the environment owner will provision them; the tester will resume after access is verified. Include the next check-in when timing affects the release decision.
Do not mark the test passed when the dependency is repaired. Repair makes execution possible; the tester still needs to perform the case and compare the actual result with the expectation.
Consider this fictional example. No Jira records or application tests are being performed here. A team plans checkout testing for release Cedar, build cedar-rc1, on its staging environment. It selects the approved checkout baseline v3, records the browser configuration, and prepares a buyer account and a tax-address dataset.
The QA lead owns the execution. Maya is assigned the payment and refund runs, while Leo is assigned the tax calculation run. Those assignments identify who should execute the work; the lead coordinates progress and decisions.
Maya discovers that the sandbox credentials are missing before she can submit a payment. She records the payment run as Blocked, notes where execution stopped, and identifies the environment owner as responsible for provisioning access. The refund scenario remains Not Run because she has not attempted it. The release summary can now distinguish an observed impediment from work still waiting to begin.
Leo reaches the tax calculation step. The case expects a tax line for the taxable cart and selected address, but the checkout displays no tax line. He records Fail, the cart and address inputs, a screenshot of the observed checkout, and a linked defect. The defect's workflow tracks investigation and repair. The failed test result records what happened on cedar-rc1.
When credentials are restored, Maya executes the payment case and records what she observes. When the tax fix arrives in cedar-rc2, Leo performs a new verification attempt against that build. Neither a repaired dependency nor a developer's completed ticket substitutes for these observations.
Preserve the earlier blocked and failed records through the app's supported attempt-history mechanism. Check its behavior before resetting a run: Xray Cloud documents that resetting a Test Run copies the current specification and discards recorded results. A fresh execution or another history-preserving approach may be more appropriate when the team needs to compare attempts. Record which case version the new attempt used, and consider related regression scope after the fix rather than assuming earlier passes still apply to the changed build.
A release summary should identify its scope and context before presenting outcomes. Separate failed, blocked, unattempted, and active work, then inspect the critical requirements behind those labels. A passed test on an earlier build or a different environment should not silently stand in for an untested change.
Review unresolved defects and blocker actions beside the results. A report showing substantial completed work can still leave the most important payment path unverified. Teams need an explicit decision about that remaining risk, not a greener-looking summary produced by removing inconvenient tests or treating blocked work as passed. Check the app's reporting definitions before comparing progress or rates across teams.
Sherlock Test Management provides one relevant version of this workflow. Its current product description connects a reviewed release-specific scope and assignments to a versioned approved baseline. Testers record Pass, Fail, or Blocked in Sherlock Runner or supported Jira panels, and failed tests can retain linked Jira defects. A team evaluating this approach can review the Sherlock product workflow. AI-proposed cases, scope, and edits still require review; these capabilities do not establish that AI has executed the application or gathered evidence automatically.
Before accepting an execution result, ask who performed the attempt, what build and case version they used, what they observed, and what remains unresolved. If a critical test is blocked, name the person who can unblock it and the condition for retesting. That gives the release owner an actionable record while keeping the Jira workflow and the testing evidence understandable on their own terms.
A green automated test history can help a developer identify a safe-looking change while leaving a QA lead unable to answer a different question:…
A checkout test appears under Payments, again in a Smoke folder, and once more under the latest release. When the payment rules change, someone…
A Jira story is Done, a test case is linked, and the related bug is Closed. Can the release owner conclude that the story's…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.
