X-Ray Integration with Robot Testing Framework - ⋮IWConnect (2023)

In this article, we'll describe the integration between the X-ray Test Management plugin for JIRA and the robot test automation framework that Jenkins uses. to help you likeintegration developer, we will work out the configurations that need to be done, so that the results of the automated execution of the tests implemented in the Robot framework can be transferred back to Xray/JIRA, where the tests are documented. All of these processes can be implemented with Jenkins work and performed without any manual intervention.

Introduction

Xray is one of the most popular test management tools for thesoftware STANDBY. Xray supports the entire test lifecycle: test planning, test design, test execution, and test reporting. Xray supports both manual and automated testing, including full compatibility with the Cucumber test. It has built-in support for various test automation platforms such as JUnit, TestNG, NUnit, Robot framework, as well as providing easy integration with CI platforms such as Jenkins, Bamboo or TeamCity. For more information about Xray, see the official documentation: https://confluence.xpand-it.com/display/XRAY/Xray+Documentation+Home
The robot framework is a popular test automation framework based on a keyword-based paradigm. The framework has a rich ecosystem around it, made up of various libraries and generic testing tools. For more details about Robot framework, refer to its official website:http://robotframework.org/.
Jenkins is the most popular open source continuous integration (CI) server, providing hundreds of plugins that allow integration with many different systems and/or frameworks, including the Xray and Robot framework.
This article is organized into three main sections: I'll introduce the first Xray test case organization into a sample project, then I'll introduce theJenkins Configurationwhich foresees the execution of the Robot test and, finally, the three different methods to import the results of the Robot's automated tests into Xray/JIRA will be elaborated.

Xray-compatible JIRA project example

Assume the system under test is e-commerce websitehttp://automationpractice.comand in Sprint 1 there are 5 user stories.

Figure 1: Sprint 1 user stories for the sample project

For each user story, tests are created in Xray. For example, for the XR-32 user story, three test cases are created in Xray:

Figure 2: X-ray tests associated with one of the sample project's user stories

The test plan is a type of custom JIRA issue provided by Xray that ensures better visibility and management of tests and can be thought of as a "wrapper" for tests related to a specific version of the system under test. We didn't go into detail on how the test plan is created (for more details on how to create a test plan using Xray, see:https://confluence.xpand-it.com/display/XRAY/Test+Plan), suppose a test plan is created with all the tests from the Sprint1 user stories:

Figure 3: Example of X-ray Test Plan

Test Run is another custom issue type provided by Xray which tracks multiple Test Runs i.e. it is made up of multiple Test Runs. From the Test Plan it is possible to create one or more Test Runs for all or just a subset of the Tests contained in the Test Plan (for example, only the Tests that still fail). We won't go into detail on how the Test Execution is created (for more details on how to create the Test Execution using Xray, see:https://confluence.xpand-it.com/display/XRAY/Test+Execution), for the sample project, a test run associated with the Sprint 1 test plan is created.

Figure 4: X-ray test run example

Installing and configuring the Xray Jenkins plugin

Xray provides a plugin for Jenkins that allows you to import automated test execution results into JIRA. The Xray Jenkins plugin can be downloaded from:https://confluence.xpand-it.com/download/attachments/35356981/xray-for-jira-connector.hpi?version=1&modificationDate=1532097682950&api=v2
To install the Xray Jenkins plugin, go to Manage Jenkins > Manage Plugins and select the Advanced tab. In the Upload Plugin section, click Upload and select the location where you downloaded the xray-for-jira-connector.hpi file.

Figure 5: Installing the Xray Jenkins plugin

Also, you need to configure the Xray plugin to communicate with JIRA. To do this, go to Manage Jenkins > Configure System > Xray Configuration for Jira. The server address field is populated with the URL of the Jira server where Xray is running, while the username and password should be the JIRA admin credentials:

Figure 6: Communication setup between Jenkins and JIRA where Xray is running

Configuring Jenkins job for automated test execution and reporting results to Xray-JIRA

The Jenkins work that will be created is a freestyle project called Xray Robot Integration Demo:

Figure 7: Create Jenkins job

For the purpose of this article, eight tests were developed in the already defined Robot for Testing on Xray framework. In this case, the custom workspace will be used and the specified directory is where the robot tests are placed.

Figure 8: Jenkins Job Configuration - Workspace Specification

To run automated tests, you can use the Windows batch command:

Figure 9: Jenkins job setup - Robot test run

And finally, to pass test execution results from Robot to Xray, like "Xray: Result Import Task" that comes with Xray Jenkins plugin, it must be set as post-build action.

Figure 10: Jenkins job configuration - post build action

After creating the Xray: Results Import Task, in the JIRA Instance field you can choose which server the JIRA results will be sent to. The Xray plugin allows you to configure multiple JIRA servers in the Xray for JIRA configuration in the Jenkins configuration. In this case, only one JIRA server is configured and the displayed JIRA instance is set as the configuration alias (see Figure 6). The Format drop-down menu lists reports from different test frameworks that can be imported into Xray.

Figure 11: X-Ray - Results Import Task

In our case, the Robot XML format will be used and the import task must be configured with the following parameters:

• Import into the same test run – When this option is checked, if you are importing multiple run report files using a global expression, the results will be imported into the same test run.
• Run report file (file path with file name) - Location and name of Robot Framework test output file. The location must be specified relative to the defined workspace
• Project key: JIRA project key. This is a required parameter, if not specified test results will not be imported into JIRA
• Test Run Key – JIRA issue key of the test run where the results will be imported. If this parameter is not specified, after importing the results, a new test run will be created in JIRA.
• Test plan key – Test plan JIRA issue key. If this parameter is not specified, the imported test run will not be associated with any test plans.
• Test environments: list of environments (separated by ",") where tests are executed.

Figure 12: Xray - Result Import Task - Configured for Robot Framework

On the other hand, the necessary configuration for the Robot tests to associate them with a certain Test in Xray/JIRA is very simple, that is, you must add a tag in the Robot test case that corresponds to the Test key in the TOUR. For example, in the test case in the figure below, "Create account" is marked with XR-2, while "Place order paid by bank transfer" is marked with XR-27, where XR-2 and XR-27 are Test keys already created. in X-rays/JIRA.

Figure 13: Association of robot test cases with test key in Xray/JIRA, using tags

In the following text, three different ways to import automated test results into Xray/JIRA will be presented.

1. Run the Jenkins job with test execution set

Let's examine what the imported robot's test results look like when in "Xray: Import Results Task" the test run key is set. To illustrate this, let's assume that before the Robot tests are executed, a Test Run has already been created in JIRA X-50, and the import task is configured as shown in Figure 12.
Once the Jenkins job completes and the bot results are imported into JIRA automatically, the test status is updated on the corresponding test run (in this case, XR-50). Also for the test environment tag it is set to the value "QA" as defined in the import task.

Figure 14: Test run created before automated test run and after robot test results are imported

2. Run the Jenkins job without a defined test run

Another option is automated bot tests that will run in Jenkins without first creating a test run in JIRA. To do this, the Test Run Key field in “Rayos: Result Import Task” must be left blank.
After the Jenkins job is complete, if we browse JIRA for the associated test plan, we can see that an additional test run (XR-51) is automatically added:

Figure 15: Test plan with the two associated test runs

3. Importing tests and test execution results from automated robots, when tests are not defined in Xray

An interesting feature of Xray is that tests can be created in JIRA based on just importing test results from Robot's automated test execution. Please note that our tests are only created in Robot and there are no corresponding tests created in Xray/JIRA.
In order for bot test cases to be assigned to a specific user story in JIRA, bot test cases must be tagged with the user story they are testing. For example, in the image below, the "Create Account" bot test case verifies the XR-1 user story, while the "Place order paid by bank transfer" and "Place order paid by check" bot tests verify the user XR-26 history.

Figure 16: Associating bot test cases with the user story key in JIRA using tags

In this case, the "minimal" configuration will be used for the import task in the post-build action in the Jenkins job, i.e. only the JIRA project environment and key will be configured, which is optional.

Figure 17: Xray - Results Import Task - Minimal configuration when tests are not created in Xray/JIRA prior to automated test execution

After Jenkins runs the tests, new tests are created, which are automatically associated with the user stories, based on the tags that were added in the Robot test cases. For example, in the case of an XR-32 user story, three Tests from the executed Robot test cases are automatically associated.

Figure 18: Tests associated and created "automatically" after the execution of the robot tests

Furthermore, after executing the tests, a new test is created with all the robot's tests executed. (Note that the test type in this case is "Generic". Xray supports three different test types: "Generic" - supports automated agnostic tests that are run by an external tool, "Cucumber" - allows you to write tests in the Gherkin language, and "Manual." - Allows you to write tests in the standard description/expected result format for manual execution)

Figure 19: Test run created after running the robot tests

Under Test Execution, you can navigate to Execution Details for each of the tests. For example, if we navigate to XR-53 Test Execution Details, we can see the test status for each of the keywords contained in the executed bot test.

Figure 20: Execution details with keywords of the executed Robot test

Comparing the three methods mentioned above for importing automated test results, it seems that method 2 is the most applicable as it automatically creates the Test Execution in Xray/JIRA when running the automated tests and associates the results with the corresponding Test Plan. On the other hand, when comparing method 2 with method 3, the disadvantage of method 3 is that it does not create any test plan associated with automated test execution. The main advantage of method 3 - automatic creation of tests in XRay/JIRA, in most situations it will not be applicable, as most likely there are tests already defined for manual tests, before the implementation and execution of automated tests.

Conclusion

X-Ray Test Management Plugin for JIRA provides all necessary artifacts for entire test lifecycle in case of manual testing, it provides strong support for automated testing with its support for importing results from different test frameworks. test automation, such as the Robot test framework. This possibility avoids the need to “manually” import the results of automated tests. Overall, Xray simplifies all test management, tracking and reporting by providing a central point – JIRA – where all test information relevant to a specific project can be found.