Playwright Jira Integration: Send Test Results Through Xray and CI

A Playwright job can produce a valid JUnit XML file while Jira shows no new test results. The report may never have been uploaded, its test identities may have changed, or the upload may have targeted a different execution. A green artifact-upload step only confirms that CI stored a file.

A dependable Playwright Jira integration separates four responsibilities: Playwright executes the tests, a reporter serializes results, CI submits the report, and Xray imports it into test-management records visible in Jira. Xray supplies the Test and Test Execution model used here. Posting a comment or attachment to an ordinary Jira issue does not create those results.

The implementation below uses Playwright Test for Node.js and Xray Cloud's v2 REST API. Data Center requires its own authentication and deployment-specific endpoint configuration. The examples describe a setup to implement and validate in your environment; they are not results from a live Jira or CI deployment.

Choose the Deployment and Execution Identity First

Before editing the pipeline, confirm that Xray is installed for the target Jira environment and that the publication identity has permission to access the project and create or update the required records. Collect the project key, an existing Test issue for an initial mapped check, and a Test Plan key if the run belongs to one. Keep credentials in CI secret storage.

For Xray Cloud, use the Xray API key's client ID and client secret to obtain a bearer token from the Xray Cloud API. A Jira Cloud API token is not a replacement for this authentication flow. For Xray Data Center, the API is hosted with Jira, uses the versioned /rest/raven/ family, and supports Jira authentication such as a Personal Access Token. Do not send Cloud credentials to a Data Center endpoint or copy a historical Server tutorial without checking the installed version.

Decide what one Test Execution represents. A useful starting policy is one CI build and one browser/environment combination per execution. Keep the stable Test issue separate from the execution that records its result. Reusing a Test key across builds preserves test identity; reusing one execution for unrelated builds can make the release record difficult to interpret.

For the examples, assume a fictional PAY project, an existing Test PAY-142, requirement PAY-98, and Test Plan PAY-90. Replace every identifier with a verified issue in your own deployment. These are illustrative keys, not working targets.

Generate XML With the Reporter Your Mapping Requires

Playwright's built-in JUnit reporter can generate ordinary XML for import. Xray-specific annotations and embedded evidence need additional handling. Xray's enhanced reporter documentation states that its extensions were supported in Playwright's built-in reporter through version 1.33 and moved to the separate package starting with version 1.34. An older tutorial that adds Xray options to the built-in reporter can therefore be misleading for a newer setup.

Configure the Enhanced Reporter Explicitly

Install the reporter in the test repository, commit the dependency and lockfile changes, and validate the selected Playwright/reporter version pair before rollout:

npm install –save-dev @xray-app/playwright-junit-reporter

In playwright.config.ts, import defineConfig from @playwright/test. A compact reporter configuration is:

export default defineConfig({ reporter: [['line'], ['@xray-app/playwright-junit-reporter', { outputFile: 'xray-report.xml', embedAnnotationsAsProperties: true }]] });

This generates a named XML file while retaining readable terminal output. Treat outputFile as a contract between the test step and the publisher. If one writes xray-report.xml and the other reads junit.xml, authentication troubleshooting will not fix the missing file.

The additional options should follow the metadata you intend to consume:

Reporter Option	Purpose	Check Before Enabling
embedAnnotationsAsProperties	Serializes ordinary annotations into testcase properties	Xray only interprets recognized property names
embedTestrunAnnotationsAsItemProperties	Maps tr: annotations into Test Run custom-field items	Corresponding Xray Test Run fields must already exist
textContentAnnotations	Writes selected annotation values as element text	Include test_description or testrun_comment when needed
embedAttachmentsAsProperty	Embeds attachments under a named evidence property	Set the value to testrun_evidence and review report size
ignoreTestCasesWithoutTestKey	Omits cases without a test_key annotation	Missing mappings can silently remove intended publication scope

The official enhanced reporter repository documents these options and their XML representation. Its upload integration is listed separately in the project's TODOs. Installing the reporter does not, by itself, submit results to Xray.

Map Tests Before Application Actions Can Fail

Without an explicit issue key or ID, Xray can match a JUnit case using its classname and name as the Generic Test Definition. If that identity has no match, import can create a new Test. Renaming a test or changing emitted identity fields can consequently produce an unexpected new issue.

For a deliberately mapped check, add the following statement inside a Playwright test callback that receives testInfo:

testInfo.annotations.push({ type: 'test_key', description: 'PAY-142' });

Place it before navigation, setup actions, and assertions that might throw. A key annotation added after a failing assertion never executes, leaving the reporter without the intended mapping. Then inspect the generated testcase and confirm that its properties contain test_key with PAY-142 as the value.

Use one clear mapping policy. If both test_id and test_key are present, Xray's Cloud JUnit documentation gives the ID precedence. An invalid explicit identity is an import error rather than a safe request to create a replacement.

Also avoid adding test_summary and test_description merely to make the report more readable. Xray can use those properties to update an existing Test's summary or description. Keep transient build notes in execution-oriented metadata unless changing the reusable Test is intentional.

Put Browser Context and Evidence in Supported Fields

An annotation named browser can exist in Playwright and appear in XML without becoming an Xray environment. Serialization and interpretation are separate operations. Choose a supported destination according to whether the information describes the execution, a test definition, or an individual Test Run.

Separate Environment Context From Test Identity

For the Cloud JUnit import endpoint, testEnvironments accepts semicolon-separated environment values. A browser-specific publication can supply staging;chromium, URL-encoded as a query parameter, together with the source revision. A separate Firefox run should carry its own environment context and report rather than inherit the Chromium label.

For a configured Test Run custom field, the reporter supports tr: annotations when embedTestrunAnnotationsAsItemProperties is enabled. For example, tr:Browser can supply a value only when Browser is an appropriate existing Xray Test Run field and the deployed import behavior supports that configuration. An arbitrary annotation does not create a field or configure Jira screens.

A requirements annotation can link the Test to existing requirement issues. That association identifies intended requirement coverage; it does not show that every acceptance criterion was tested. Review the assertion and test data against the requirement, especially after a behavior change.

Attach Evidence Deliberately

Use Playwright's attachment API to associate evidence with the current test. For example, after obtaining a screenshot buffer from page.screenshot(), attach it using:

await testInfo.attach('checkout.png', { body: screenshot, contentType: 'image/png' });

With embedAttachmentsAsProperty set to testrun_evidence, the enhanced reporter can embed attachment contents as Base64 items in the XML. A filesystem path in CI is not automatically an attachment accessible from Jira. Inspect the report for the evidence item, then inspect the imported Test Run for the actual file.

Embedding evidence enlarges the report and may expose customer data, tokens, or payment details captured on screen. Prefer focused, sanitized failure evidence and retain richer traces in controlled CI artifact storage when appropriate. Configure failure hooks so that an early exception does not prevent the evidence you need from being collected.

Publish Results as a Distinct CI Operation

The initial pipeline should make test failure and publication failure independently visible. Otherwise, a failed assertion can skip the publisher, or a successful upload can obscure a failed suite. Keep the test command's failure status while allowing diagnostic and publication steps to run when a report exists.

For GitHub Actions, an appropriate publication step can use the condition !cancelled(), together with a check that the expected report exists and is usable. This allows it to run after a test failure without treating a canceled job as a normal complete run. Protect the publication credentials separately from untrusted pull-request execution.

Use this sequence for the first browser/environment path:

1. Install the lockfile-resolved dependencies with npm ci, then install browsers and system dependencies with npx playwright install –with-deps on the selected CI agent.
2. Run npx playwright test with the configured reporters and intended project selection. Record the command's outcome without converting a failed test run into success.
3. Preserve xray-report.xml and the native Playwright report as CI artifacts. Verify that XML generation completed and that mapped cases and expected evidence are present.
4. Authenticate to the correct Xray deployment in a trusted publication step. For Cloud v2, exchange client_id and client_secret for a token using POST /api/v2/authenticate on the Xray Cloud API host.
5. Submit the XML to the Cloud JUnit import endpoint with the intended project, execution policy, environment, and revision. Record the import outcome separately from the suite outcome.
6. Read the returned execution identity, open that execution in Jira/Xray, and compare the imported Test Runs with the report. Retain the CI build reference and execution key together.

Playwright recommends a globalTimeout so the runner can stop and finish reporting before the CI platform kills a hung job. If you also configure a CI job timeout, leave enough time afterward for report handling and publication. A terminated process may leave no complete report to import.

Use the Cloud API Contract Without Mixing Payload Types

The Cloud host is https://xray.cloud.getxray.app. Authentication uses a JSON object containing client_id and client_secret; the successful response is a JSON string containing the token. Decode that string before constructing the Authorization: Bearer header. Do not print the token, enable shell tracing around it, or store authentication responses in ordinary build artifacts.

The standard upload request is POST /api/v2/import/execution/junit. Send the XML file as the request body with an XML content type, such as text/xml, and the bearer header. Supply projectKey=PAY for the fictional target project and, when applicable, testPlanKey=PAY-90. Add the relevant testEnvironments and revision values using a query-parameter encoder rather than assembling unescaped strings.

Omitting testExecKey allows Xray to create a new Test Execution. Supplying testExecKey targets an existing execution. These are different lifecycle choices: use a known execution deliberately, not as a blanket workaround for duplicate imports. The stable PAY-142 Test key inside the XML does not identify the Test Execution.

If you need detailed issue-field control, Xray also documents a JUnit multipart endpoint with separate results and metadata parts. Do not send a multipart tutorial's metadata body to the plain XML endpoint. Field IDs, required fields, issue types, and workflow permissions must match the actual Jira configuration.

A CI plugin or action can wrap the same publication process. Check its deployment selector, supported report format, authentication inputs, report path, and returned execution identity. Choosing an action with Jira in its name does not establish that it imports Xray Test Runs.

Keep Publication Secrets Out of Untrusted Test Jobs

GitHub withholds ordinary repository secrets from fork pull-request workflows. Treat that as a trust boundary, rather than a reason to expose Xray credentials to arbitrary test code. Publish from an approved trusted context, or separate restricted publication from untrusted execution with careful artifact validation.

Do not switch to pull_request_target and then execute the fork's code with privileged secrets. A report from another workflow is also untrusted input. Restrict its origin, size, and intended target before a credentialed publisher processes it.

Verify Imported Records Before Scaling the Pipeline

Start with a controlled pair of checks in a test project: one that passes and one that fails. This illustrative validation deliberately tests publication behavior, not application readiness. For each check, compare its XML identity and outcome with the resulting Xray record.

A useful acceptance check covers all of the following:

• The import response is successful and contains the Test Execution identity expected for that deployment.
• Existing mapped Test keys are used, with no unintended new Test issues.
• Pass and failure outcomes appear on the intended Test Runs, with failure details available.
• Browser/environment, source revision, and optional Test Plan associations identify the intended run.
• Evidence files are accessible, and the CI suite remains failed when an assertion failed.

If no execution appears, check whether the publication step ran before investigating Xray mapping. If it ran, inspect the exact report path, authentication response status, upload status, and sanitized response body. An XML file in the CI artifact list proves none of these downstream steps.

If unexpected Tests appear, inspect testcase classname/name and explicit key properties. If browser metadata is missing, compare the import parameters and configured Test Run fields with the emitted XML; adding more arbitrary annotations will not configure Xray. If evidence is absent, determine whether it was attached in Playwright, embedded by the reporter, and accepted during import.

Avoid blindly replaying an import after a timeout. The server may have created an execution even when CI lost the response. Reconcile the build and target execution before retrying a request that could create another record.

Retries and sharding need another validation pass. Playwright distinguishes a first-attempt pass from a flaky test that passes on retry; confirm what survives the chosen reporter and Xray import. Retain the native report for investigation. With shards, use separate artifact names and an intentional aggregation/publication design. Concatenating XML documents is not a valid merge strategy, and parallel publishers should not overwrite the same file or race against one shared execution without a verified policy.

Keep Requirement Review Separate From Result Transport

A correctly imported Playwright failure tells the team what happened for a particular test and build. It does not establish that the suite still covers a changed checkout rule. Review reusable test coverage when acceptance criteria change, then decide which automated and human-executed checks belong in the release.

For that review work, Sherlock Test Management describes AI-proposed cases derived from Jira issues and other product context, human approval into a versioned baseline, and release-specific test scope. A team reviewing a new checkout permission rule can assess proposed cases before adding them to its approved coverage. Those design and review capabilities do not establish Playwright result ingestion, Xray interoperability, or automated browser execution in Sherlock; the transport path in this tutorial is specifically through Xray.

Roll out the integration after a failed test remains visible as a failed CI run and as the correctly mapped Xray result, with usable evidence and an identifiable build. Expand to more browsers and shards only after that small path is reliable. The acceptance criterion is a result the release team can trace and interpret, rather than the existence of an XML file.