A checkout test fails in Chrome on Windows, then passes in Firefox on the same operating system. The Windows summary turns green. Before treating that as a fix, check what the summary groups together: the later Firefox result can describe Windows while leaving the Chrome failure unresolved.
Xray Cloud Test Environments help teams reuse tests and compare execution results across environments. Their usefulness depends on the question encoded in the environment names. Separate browser and operating-system values support dimension-level analysis; they do not preserve every browser/OS pair as an independently reportable result. A release decision needs a defined matrix as well as an understanding of the summary.
In Xray, the Test holds the reusable case. A Test Execution is a Jira issue that brings selected Tests together for execution in a target context. The association between a Test and that Test Execution is a Test Run, which holds that execution's result. A Test Run is not another Jira issue.
For a checkout case with the same steps and expected outcome across browsers, retain one Test and organize its executions by the context being tested. Avoid creating browser-specific copies merely to store separate outcomes: copies introduce another place for steps to drift when checkout behavior changes. A genuinely different expected behavior still needs an explicit case-design decision.
Test Environments belong to the execution layer. Naming Chrome and Windows describes where the work is scheduled or recorded; it does not launch Chrome, install Windows, reserve a machine, or verify the deployed application. A tester or a separately configured automation system must perform the test. Treat the Test, execution record, actual browser session, and attached evidence as related but distinct artifacts. These are Xray app concepts within Jira, not browser-execution capabilities supplied by Jira itself.
Decide whether a report primarily needs to answer “How is Chrome doing?” or “Did this exact case pass on Chrome and Windows?” The two naming models retain different information for Xray's environment-based analysis. Adding more names without choosing the reporting question can produce a larger namespace without resolving the evidence gap.
Xray's Cloud guide recommends treating dimensions individually. For an execution performed in Firefox on macOS, assign Firefox and macOS as separate Test Environment values. Those two values describe one execution context; they do not mean that two browser sessions ran.
The namespace grows by the sum of the values in each dimension. With three browsers and two operating systems, there are five dimension values rather than six composed pairs. Adding two testing stages would give seven values rather than twelve three-part names. These counts describe names to manage, not the amount of testing required.
This model supports analysis from a browser perspective or an operating-system perspective. The documented limitation is that you cannot analyze the combined Firefox-and-macOS environment as a joint tuple through those separate values. A Firefox summary may draw on Firefox executions from more than one operating system. Preserve the exact pair in execution evidence when that pair is important to acceptance.
A composed environment name gives a pair its own identity, such as Chrome-on-Windows11. This can express a specific tuple, but the guide says that coverage analysis is then restricted to that composed environment rather than either variable independently. Do not assume that a collection of composed names also supplies an independent Chrome rollup.
Xray discourages composed environment names because they pollute the namespace and make management harder. It also warns against dozens or hundreds of environments because usage becomes harder and performance overhead increases. The guide does not give a safe numerical threshold for your project.
Decision | Separate Browser and OS Values | Composed Pair Name |
|---|---|---|
Reporting unit | One dimension value | One named tuple |
Joint browser/OS analysis | Not available from separate values | Tuple has its own environment identity |
Independent browser or OS view | Supported dimension perspective | Not supplied by the composed name |
Namespace in the three-browser, two-OS example | Five values | Six names |
Vendor guidance | Recommended for multiple dimensions | Discouraged because management scales poorly |
Use separate values when independent browser or OS views meet the main reporting need. If pair-level acceptance is mandatory, keep an explicit case-by-pair record and decide how it will be maintained. Composed names require accepting the documented costs; they are not a general upgrade to a native full Cartesian-matrix report. Adding both kinds of name indiscriminately introduces more environment identities into result consolidation and needs a deliberate reporting design.
Consider an illustrative release, not a customer case or an observed Jira tenant. Candidate checkout-4.2-rc1 belongs to version 4.2 and the Checkout 4.2 Test Plan. CHECK-101 tests successful checkout submission; CHECK-102 tests checkout validation. The team's fictional support policy includes Chrome, Firefox, and Edge on both Windows 11 and macOS 14.
That policy creates six browser/OS pairs and twelve required case-pair cells. Assume the following six records are the only runs that exist, all completed on that same candidate. The remaining planned cells have not been scheduled as additional TODO runs; they are gaps in the planning matrix, not extra inputs to the aggregation example. Every result is final PASS or FAIL, creation order matches completion order, and there are no timestamp ties or custom statuses. The named browser and OS values are assumed to reflect the actual fictional session, not an automatic measurement.
Completion Time | Test | Actual Browser | Actual OS | Result |
|---|---|---|---|---|
09:00 | CHECK-101 | Chrome | Windows 11 | FAIL |
09:15 | CHECK-102 | Chrome | Windows 11 | PASS |
09:30 | CHECK-101 | Firefox | Windows 11 | PASS |
10:00 | CHECK-101 | Firefox | macOS 14 | PASS |
10:15 | CHECK-102 | Firefox | macOS 14 | PASS |
10:30 | CHECK-101 | Chrome | macOS 14 | PASS |
Before 10:30, CHECK-101 has a passing latest Windows 11 result from Firefox, but its Chrome result remains FAIL. In the guide's simple final-status model, All Environments remains failed because the latest Chrome result failed. The newer Firefox pass did not replace the Chrome label's history.
After 10:30, the latest Chrome result is PASS from macOS. The latest Firefox, Windows 11, and macOS 14 results are also PASS. All four represented dimension values can therefore be green while the recorded Chrome/Windows 11 pair still has its earlier failure. A passing Chrome result on macOS is useful evidence about that context; it is not a rerun of the failing Windows combination.
Edge has no recorded execution in this scenario. Its two pairs for CHECK-101 remain untested, regardless of the green represented labels. Across both cases, six of the twelve required cells have completed records: five PASS and one FAIL. The other six cells have no recorded execution. A label-level summary cannot establish completion of those missing cells.
Read the case-pair record separately from the dimension summary:
Browser | CHECK-101 on Windows 11 | CHECK-101 on macOS 14 | CHECK-102 on Windows 11 | CHECK-102 on macOS 14 |
|---|---|---|---|---|
Chrome | FAIL | PASS | PASS | No Record |
Firefox | PASS | PASS | No Record | PASS |
Edge | No Record | No Record | No Record | No Record |
No Record means that this illustrative record contains no execution for that required cell. It is not a claim about how Xray automatically represents an uncreated run. This planning view keeps the old Chrome/Windows failure visible even when dimension summaries have moved on.
Keep the failing cell visible until a matching rerun supplies new evidence or an accountable release decision explicitly accepts the outstanding risk. Preserve the earlier failure and its defect/evidence trail even after a later rerun passes. Removing an environment from a view changes the question being asked, not the historical result.
A Test's consolidated status is calculated in a context. Xray's Coverage Analysis guide distinguishes Latest, Version, and Test Plan scopes, with Test Environment and final-status precedence affecting analysis. Record those choices with the report so another reviewer can reproduce the question.
For version analysis, Xray considers the Test Execution's Fix Version. For Test Plan analysis, it considers Test Executions linked to that plan. A story's version alone is not enough to establish that the relevant executions belong to the intended version. Confirm the relationships rather than comparing two green screenshots with different filters. A version or plan can also include attempts against different candidate builds. Those scope labels do not replace an exact artifact identity in the evidence.
The Cloud calculation guide documents another important choice for Version or Test Plan analysis: with final-status precedence enabled, final results take priority and the latest run is selected using its finished-on date. With the option disabled, selection uses the Test Run's created date. The guide also shows that an environment containing only a non-final run can still contribute that run. Do not interpret precedence as a guarantee that every unfinished environment disappears.
Those rules explain why “latest” should not be reduced to the newest Test Execution issue number or the most recently edited Jira issue. The worked example avoids ordering ambiguity by aligning creation and completion order and using only final PASS/FAIL results. For real reports, verify the chosen scope, status configuration, and relevant run dates. Xray notes that environment-selection behavior depends on the report, so do not transfer an assumption between reports without checking it.
An empty Test Environment is also significant. The guide treats it similarly to a named environment. A passing run without a browser label does not replace a failing Chrome run as evidence for Chrome. Investigate unlabeled executions instead of treating the blank value as “all browsers.”
A family-level label such as Chrome is convenient for analysis but too coarse to reproduce a browser failure. Capture the specific browser version and OS build, candidate artifact, endpoint, relevant configuration, and test-data context through your team's approved evidence process. These are recommended evidence fields, not a claim that Xray automatically collects them.
Before execution, reconcile the intended context with the available session. Useful run-specific evidence includes:
An environment label also says little about whether the deployed build, dependencies, or shared test data are ready. Keep that start decision separate from the case outcome. The test environment readiness checks provide a useful companion for recording those prerequisites and rechecking them when conditions change.
The same separation applies when maintaining cases in another Jira testing workflow. Sherlock Test Management describes approved versioned baselines, reviewed release-specific scope, and Pass, Fail, or Blocked recording through its Runner or supported Jira panels. For checkout work, this separates the approved case version from the release attempt and its evidence. It does not establish a Sherlock-Xray integration, automatic browser-matrix aggregation, or browser provisioning. Whichever app stores results, retain enough context to connect the outcome to the case version and the environment actually used.
Use the naming model to support analysis, then make the acceptance decision against the required work:
For the checkout example, record the corrected candidate identity and rerun CHECK-101 on Chrome/Windows 11 after the suspected fix. Then complete the six missing cells, or have the release decision owner explicitly accept each remaining gap and its consequence. A new build also needs a reasoned decision about which earlier results remain relevant; do not silently carry every pass forward. A green browser or OS summary is useful for triage. Pair-level acceptance still needs evidence for the pair the team promised to test.
A green automated test history can help a developer identify a safe-looking change while leaving a QA lead unable to answer a different question:…
A checkout test appears under Payments, again in a Smoke folder, and once more under the latest release. When the payment rules change, someone…
A Jira story is Done, a test case is linked, and the related bug is Closed. Can the release owner conclude that the story's…
Create traceable test cases, prepare release-specific test runs, and keep approved coverage current as the product changes.
