Piloting automated tests executions from Squash TM
Info
To pilote automated tests executions from Squash TM, following components are required:
- Squash TM
- Squash Orchestrator
- Result Publisher Plugin for Squash TM
- Squash AUTOM plugin for Squash TM
- In case of Agilitest or UFT test execution: an OpenTestFactory agent for Agilitest/UFT execution environment
- In case of Ranorex test execution: an OpenTestFactory agent for Ranorex execution environment, and an environment variable called SQUASH_MSBUILD_PATH containing the path to the parent folder of MSBuild.exe
Squash TM test case automation
Info
This page describes the operations common to all supported test frameworks in this version. You can access the automation specificities for each technology directly with the following links:
Without using the Squash automation workflow
To execute a test case using the Squash Orchestrator, its Automation block in the Information tab of the test case page must be correctly filled:
-
Automated test technology
: A dropdown list allowing you to choose the technology used for the execution of a test case. In this version, only Robot Framework, Junit, Cucumber, Cypress, SoapUi, Agilitest and Ranorex are functioning. -
Source code repository URL
: The address of the source code repository where the project is located, as referenced in the Source code management servers area of the Administration. -
Automated test reference
: This is the location of the automated test within the project. This reference must follow the specific format of the used test technology (see here).
Info
Agilitest, Ranorex and UFT are only supported by the Premium version of Squash AUTOM.
Using the Squash automation workflow
Regular test case
To execute a test case using the Squash Orchestrator, it must be automated in the Automation Workspace by filling three columns:
-
Auto. test tech.
: A dropdown list allowing you to choose the technology used for the execution of a test case. -
Scm URL
: The address of the source code repository where the project is located. -
Auto. test ref.
: This is the location of the automated test within the project. This reference must follow the specific format of the used test technology (see here).
-
TECHNO.
: A dropdown list allowing you to choose the technology used for the execution of a test case. -
SCM URL
: The address of the source code repository where the project is located. -
REF. OF THE AUTO TEST
: This is the location of the automated test within the project. This reference must follow the specific format of the used test technology (see here).
Info
Agilitest, Ranorex and UFT are only supported by the Premium version of Squash AUTOM.
BDD or Gherkin test case
The information of the Automation block is automatically filled during the transmission of a BDD or Gherkin script to a remote source code repository hosting service. It can also be modified by the user at any moment.
Squash TM parameters exploitation
When a Squash TM execution plan is launched (through a PEaC or directly from the campaign workspace), Squash TM will transmit various information on execution plan item that can be exploited by a test case. Details of this functionality can be found on the corresponding used technology section.
Info
Following frameworks can handle Squash TM parameters exploitation during test execution:
Automation frameworks specificities
Automation with Agilitest
This feature is available only in the Premium version of Squash AUTOM.
1. Test reference
In order to bind a Squash TM test case to an Agilitest automated test, the content of the Automated test reference field of the Automation block of a test case must have the following format:
[1] / [2]
With:
-
[1]
: Name of the project on the source code repository. -
[2]
: Path and name of the ActionTestScript file, from the root of the project (with the.ats
extension).
Focus
The ATS script must be located in src/main/ats/*
, as in any regular ATS project architecture.
2. Nature of the exploitable Squash TM parameters
The exploitable Squash TM parameters in an ActionTestScript script will differ depending on whether you're using the Community or Premium version of Squash DEVOPS.
Here is a table showing the exploitable parameters:
Nature | Key | Community | Premium |
---|---|---|---|
Name of the dataset | DSNAME | ||
Dataset parameter | DS_[name] | ||
Test case reference | TC_REF | ||
Test case custom field | TC_CUF_[code] | ||
Iteration custom field | IT_CUF_[code] | ||
Campaign custom field | CPG_CUF_[code] | ||
Test suite custom field | TS_CUF_[code] |
Legend:
[code]
: Value of the "Code" of a custom field[name]
: Parameter name as filled in Squash TM
3. Squash TM parameters usage
When executing a Squash TM automated test case with Agilitest, it is possible to exploit the Squash TM parameters inside the test.
In order to achieve this, you'll have to follow these steps:
-
Create custom fields in Squash TM and bind them to the project bearing the test plan to execute.
-
Make sure that the code fields of the parameters correspond to the names of the existing environment variables present in the ActionTestScript script.
Info
Squash TM adds a prefix to the code of the transmitted custom field. Make sure to take it into account. Please refer to the Squash TM documentation for more information.
Below is an example of an Agilitest test file and the corresponding Squash TM test case automation:
4. Supported versions
Squash AUTOM and Squash DEVOPS have been validated with ATS 2.1.4. Any recent version should work properly.
Automation with Cucumber
1. Test reference
Info
If a scenario name is not specified, the result of each executed Squash TM test case is calculated by taking into account the individual results of each scenario included in the bound file:
-
If at least one scenario has an Error status (in case of a technical issue), the status of the execution will be Blocked.
-
If at least one scenario fails functionally and none of the other has an Error status, the status of the execution will be Failed.
-
If all scenarios succeed, the status of the execution will be Success.
In order to bind a Squash TM test case to a Cucumber automated test, the content of the Automated test reference field of the Automation block of a test case must have the following format:
[1] / [2] # [3] # [4]
With:
-
[1]
: Name of the project on the source code repository. -
[2]
: Path and name of the Cucumber test file, from the root of the project (with the.feature
extension). -
[3]
: feature name as specified in the Cucumber test file. This parameter is optional. -
[4]
: scenario name as specified in the Cucumber test file. This parameter is optional.
2. Nature of the exploitable Squash TM parameters
Squash AUTOM and Squash DEVOPS are able to use the name of a Squash TM dataset as a tag value to use for the execution of a specific subset of a Cucumber feature.
Both Community and Premium versions can use dataset names.
3. Squash TM parameters usage
When executing a Squash TM automated test case with Cucumber, it is possible to exploit the Squash TM dataset name in order to execute a specific dataset of a Cucumber scenario.
In order to achieve this, you'll have to follow these steps:
-
Fill the datasets in the Parameters tab of the test case in Squash TM.
-
Create in a Cucumber scenario as many example table as there are dataset in Squash TM test case. Annotate them with a tag corresponding to the name of a Squash TM dataset.
-
Create one line of elements in each example table to set scenario's parameters values for the dataset.
Below is an example of a Cucumber test file and the corresponding Squash TM test case automation:
4. Supported versions
Squash AUTOM and Squash DEVOPS have been validated with Cucumber-JVM 4.2.6. Some users reported some problems with Cucumber-JVM 6.X, these ones will be analyzed.
Automation with Cypress
1. Test reference
Info
In this version of Squash AUTOM, it is not possible to select a specific test in a .spec.js
file containing several ones: all tests in the file are therefore executed together. The result of each executed Squash TM test case is calculated by taking into account the individual results of each test included in the bound file:
-
If at least one test has an Error status (in case of a technical issue), the status of the execution will be Blocked.
-
If at least one test fails functionally and none of the other has an Error status, the status of the execution will be Failed.
-
If all tests succeed, the status of the execution will be Success.
In order to bind a Squash TM test case to a Cypress automated test, the content of the Automated test reference field of the Automation block of a test case must have the following format:
[1] / [2]
With:
-
[1]
: Name of the project on the source code repository. -
[2]
: Path and name of the Cypress test file, from the root of the project (with the.spec.js
extension).
2. Nature of the exploitable Squash TM parameters
The exploitable Squash TM parameters in a Cypress script will differ depending on whether you're using the Community or Premium version of Squash DEVOPS.
Here is a table showing the exploitable parameters:
Nature | Key | Community | Premium |
---|---|---|---|
Name of the dataset | DSNAME | ||
Dataset parameter | DS_[name] | ||
Test case reference | TC_REF | ||
Test case custom field | TC_CUF_[code] | ||
Iteration custom field | IT_CUF_[code] | ||
Campaign custom field | CPG_CUF_[code] | ||
Test suite custom field | TS_CUF_[code] |
Legend:
[code]
: Value of the "Code" of a custom field[name]
: Parameter name as filled in Squash TM
3. Squash TM parameters usage
When executing a Squash TM automated test case with Cypress, it is possible to exploit the Squash TM parameters inside the test.
In order to achieve this, you'll have to follow these steps:
-
Create custom fields in Squash TM and bind them to the project bearing the test plan to execute.
-
Make sure that the code fields of the parameters correspond to the names of the existing environment variables present in the Cypress script.
Info
Squash TM adds a prefix to the code of the transmitted custom field. Make sure to take it into account.
Please refer to the Squash TM documentation for more information.
Below is an example of a Cypress test file and the corresponding Squash TM test case automation:
4. Supported versions
Squash AUTOM and Squash DEVOPS have been validated with Cypress 8.5.0. Any recent version should work properly.
Automation with JUnit
1. Test reference
In order to bind a Squash TM test case to a JUnit automated test, the content of the Automated test reference field of the Automation block of a test case must have the following format:
[1] / [2] # [3]
With:
-
[1]
: Name of the project on the source code repository. -
[2]
: Qualified name of the test class. -
[3]
: Name of the method to test in the test class. This parameter is optional.
Below is an example of a test class and the corresponding Squash TM test case automation:
2. Nature of the exploitable Squash TM parameters
The exploitable Squash TM parameters in a JUnit script will differ depending on whether you're using the Community or Premium version of Squash DEVOPS.
Here is a table showing the exploitable parameters:
Nature | Key | Community | Premium |
---|---|---|---|
Name of the dataset | DSNAME | ||
Dataset parameter | DS_[name] | ||
Test case reference | TC_REF | ||
Test case custom field | TC_CUF_[code] | ||
Iteration custom field | IT_CUF_[code] | ||
Campaign custom field | CPG_CUF_[code] | ||
Test suite custom field | TS_CUF_[code] |
Legend:
[code]
: Value of the "Code" of a custom field[name]
: Parameter name as filled in Squash TM
3. Squash TM parameters usage
It is possible when running an automated Squash TM test case with JUnit to exploit Squash TM parameters within it.
To do this, the following steps must be followed:
-
Fill in custom fields in Squash TM and associate them with the project containing the test plan to be executed.
-
Import opentestfactory-java-param-library into the JUnit project containing the tests to run by adding to the
pom.xml
file:- the following Maven repository:
<repositories> <repository> <id>org.squashtest.repo.release</id> <name>Squashtest repository for releases</name> <url>https://repo.squashtest.org/maven2/release</url> </repository> </repositories>
- the following Maven dependency:
<dependencies> <dependency> <groupId>org.opentestfactory.util</groupId> <artifactId>opentestfactory-java-param-library</artifactId> <version>1.0.0</version> </dependency> </dependencies>
-
You can then retrieve the value of a Squash TM parameter of the desired type using the following syntax:
ParameterService.INSTANCE.getString("paramName");
ParameterService.INSTANCE.getInt("paramName");
ParameterService.INSTANCE.getFloat("paramName");
ParameterService.INSTANCE.getDouble("paramName");
ParameterService.INSTANCE.getBoolean("paramName");
-
The above methods look for the desired parameter in the test parameters transmitted by Squash TM; if they cannot find it, they then look for it in the global parameters transmitted by Squash TM.
These methods throw aParameterNotFoundException
if the parameter is not found. If the parameter is found but cannot be converted to the desired type, aParameterFormatException
is thrown. Consider handling these exceptions in your test classes.
Warning: the conversion of Boolean data returnstrue
when the character string to be converted is equal to"true"
(whatever the case),false
in all other cases; but never propagates an exception. -
It is also possible to define a default parameter in the case where the Squash TM parameter is non-existent by using the following syntax:
ParameterService.INSTANCE.getString("paramName", defaultValue);
ParameterService.INSTANCE.getInt("paramName", defaultValue);
ParameterService.INSTANCE.getFloat("paramName", defaultValue);
ParameterService.INSTANCE.getDouble("paramName", defaultValue);
ParameterService.INSTANCE.getBoolean("paramName", defaultValue);
-
The above methods therefore do not propagate a
ParameterNotFoundException
when the Squash TM parameter sought is not found but propagate aParameterFormatException
if the Squash TM parameter is found, but cannot be converted to the desired type. -
It is also possible to target a test parameter or a global parameter with specific methods. As with the previous methods, they are available in versions with and without default parameters. Here are a few examples:
ParameterService.INSTANCE.getTestString("paramName");
ParameterService.INSTANCE.getGlobalInt("paramName");
ParameterService.INSTANCE.getTestFloat("paramName", defaultValue);
ParameterService.INSTANCE.getGlobalBoolean("paramName", defaultValue);
Below is an example of a JUnit test file and the automation of the associated Squash TM test case:
4. Supported versions
Squash AUTOM and Squash DEVOPS have been validated with
- JUnit 4.12, any JUnit 4 version more recent should work properly
- JUnit 5.3.2, any JUnit 5 version should work properly
Additionally, it is advised to use Surefire 2.19.1 or later.
Automation with Ranorex
This feature is available only in the Premium version of Squash AUTOM.
In order to use this feature, an OpenTestFactory agent for Ranorex execution environment is needed. Furthermore, an environment variable called SQUASH_MSBUILD_PATH containing the path to the parent folder of MSBuild.exe must be created. You can execute the following command to get the path to your MSBuild executable:
reg.exe query "HKLM\SOFTWARE\Microsoft\MSBuild\ToolsVersions\4.0" /v MSBuildToolsPath
1. Test reference
In order to bind a Squash TM test case to a Ranorex automated test, the content of the Automated test reference field of the Automation block of a test case must have the following format:
[1] / [2] # [3] # [4] # [5]
With:
-
[1]
: Name of the project on the source code repository. -
[2]
: The path to the solution's.sln
file from the root of the project source repository. -
[3]
: Ranorex project name to execute. -
[4]
: Ranorex test suite name to execute. -
[5]
: Ranorex test case name to execute. This parameter is optional.
2. Nature of the exploitable Squash TM parameters
Here is a table showing the exploitable parameters:
Nature | Key |
---|---|
Name of the dataset | DSNAME |
Dataset parameter | DS_[name] |
Test case reference | TC_REF |
Test case custom field | TC_CUF_[code] |
Iteration custom field | IT_CUF_[code] |
Campaign custom field | CPG_CUF_[code] |
Test suite custom field | TS_CUF_[code] |
Legend:
[code]
: Value of the "Code" of a custom field[name]
: Parameter name as filled in Squash TM
3. Squash TM parameters usage
When executing a Squash TM automated test case with Ranorex, it is possible to exploit the Squash TM parameters inside the test.
In order to achieve this, you'll have to follow these steps:
-
Create custom fields in Squash TM and bind them to the project bearing the test plan to execute.
-
Make sure that the code fields of the parameters correspond to the names of the existing parameters present in the Ranorex solution.
Info
Squash TM adds a prefix to the code of the transmitted custom field. Make sure to take it into account. Please refer to the Squash TM documentation for more information.
Below is an example of a Ranorex solution and the corresponding Squash TM test case automation:
4. Supported versions
Squash AUTOM and Squash DEVOPS have been validated with Ranorex 9.5. Any recent version should work properly.
Automation with Robot Framework
1. Test reference
In order to bind a Squash TM test case to a Robot Framework automated test, the content of the Automated test reference field of the Automation block of a test case must have the following format:
[1] / [2] # [3]
With:
[1]
: Name of the project on the source code repository.[2]
: Path and name of the Robot Framework test, from the root of the project (with the.robot
extension).[3]
: Name of the test case to execute in the.robot
file.
2. Nature of the exploitable Squash TM parameters
The exploitable Squash TM parameters in a Robot Framework script will differ depending on whether you're using the Community or Premium version of Squash DEVOPS.
Here is a table showing the exploitable parameters:
Nature | Key | Community | Premium |
---|---|---|---|
Name of the dataset | DSNAME | ||
Dataset parameter | DS_[name] | ||
Test case reference | TC_REF | ||
Test case custom field | TC_CUF_[code] | ||
Iteration custom field | IT_CUF_[code] | ||
Campaign custom field | CPG_CUF_[code] | ||
Test suite custom field | TS_CUF_[code] |
Legend:
[code]
: Value of the "Code" of a custom field[name]
: Parameter name as filled in Squash TM
3. Squash TM parameters usage
When executing a Squash TM automated test case with Robot Framework, it is possible to exploit the Squash TM parameters inside the test.
In order to achieve this, you'll have to follow these steps:
-
Create custom fields in Squash TM and bind them to the project bearing the test plan to execute.
-
Install the squash-tf-services python library on the environment where the Robot Framework execution takes place. It is accessible through the
pip
package management and can be installed by executing the following command line:
python -m pip install squash-tf-services
- Import the library inside the
.robot
file in the Settings section:
Library squash_tf.TFParamService
- You can then retrieve the value of a Squash TM parameter by calling the following keyword:
Get Param <parameter key>
Below is an example of a Robot Framework test file and the corresponding Squash TM test case automation:
4. Supported versions
Squash AUTOM and Squash DEVOPS have been validated with Robot Framework 4.0. Any recent version should work properly.
Automation with SKF
1. Test reference
In order to bind a Squash TM test case to a SKF automated test, the content of the Automated test reference field of the Automation block of a test case must have the following format:
[1] / [2] . [3] # [4]
With:
-
[1]
: Path to the root SKF folder (which contains the pom.xml file) on the source repository. -
[2]
: Default test ecosystem of the SKF project (tests). -
[3]
: Child tests ecosystem (it is possible to add several by separating them by.
; this parameter is optional). -
[4]
: Name of the test script to run (with its.ta
extension; this parameter is mandatory)
2. Nature of the exploitable Squash TM parameters
The exploitable Squash TM parameters in a SKF script will differ depending on whether you're using the Community or Premium version of Squash DEVOPS.
Here is a table showing the exploitable parameters:
Nature | Key | Community | Premium |
---|---|---|---|
Name of the dataset | DSNAME | ||
Dataset parameter | DS_[name] | ||
Test case reference | TC_REF | ||
Test case custom field | TC_CUF_[code] | ||
Iteration custom field | IT_CUF_[code] | ||
Campaign custom field | CPG_CUF_[code] | ||
Test suite custom field | TS_CUF_[code] |
Legend:
[code]
: Value of the "Code" of a custom field[name]
: Parameter name as filled in Squash TM
3. Squash TM parameters usage
When executing a Squash TM automated test case with SKF, it is possible to exploit the Squash TM parameters inside the test.
In order to achieve this, you'll have to follow these steps:
-
Create custom fields in Squash TM and bind them to the project bearing the test plan to execute.
-
Call desired parameters inside a file used by the SKF test by using
${key}
placeholder. -
In the SKF test, use the command to replace, in a file, parameters' placeholders by values transmitted at execution. The command has the following syntax:
CONVERT {resourceToConvert<Res:file>} TO file (param) USING context_global_params, context_script_params AS {converted<Res:file>}
Info
Learn more about usage of context parameters in a SKF test in the dedicated documentation
Below is an example of a SKF project using a parameter in a xml file and the corresponding Squash TM test case automation:
4. Supported versions
Squash AUTOM and Squash DEVOPS have been validated with SKF 1.14.0.
Automation with SoapUI
Test reference
In order to bind a Squash TM test case to a SoapUI automated test, the content of the Automated test reference field of the Automation block of a test case must have the following format:
[1] / [2] # [3] # [4]
With:
-
[1]
: Name of the project on the source code repository. -
[2]
: Path and name of the SoapUI test file, from the root of the project (with the.xml
extension). -
[3]
: Name of the TestSuite containing the test case. -
[4]
: Name of the test case to execute.
Below is an example of a SoapUI test file and the corresponding Squash TM test case automation:
4. Supported versions
Squash AUTOM and Squash DEVOPS have been validated with SoapUI 5.6.0. Any recent version should work properly.
Automation with UFT
This feature is available only in the Premium version of Squash AUTOM.
In order to use this feature, an OpenTestFactory agent for UFT execution environment is needed.
Warning
Squash AUTOM/Squash DEVOPS current version does not support UFT Service Tests.
Focus
For result publication to Squash TM functionnality, UFT must be configured to generate RRV reports
1. Référence du test
In order to bind a Squash TM test case to a UFT automated test, the content of the Automated test reference field of the Automation block of a test case must have the following format:
[1]
With:
[1]
: The path to the test folder to execute.
2. Nature of the exploitable Squash TM parameters
Here is a table showing the exploitable parameters:
Nature | Key |
---|---|
Name of the dataset | DSNAME |
Dataset parameter | DS_[name] |
Test case reference | TC_REF |
Test case custom field | TC_CUF_[code] |
Iteration custom field | IT_CUF_[code] |
Campaign custom field | CPG_CUF_[code] |
Test suite custom field | TS_CUF_[code] |
Legend:
[code]
: Value of the "Code" of a custom field[name]
: Parameter name as filled in Squash TM
3. Squash TM parameters usage
When executing a Squash TM automated test case with UFT, it is possible to exploit the Squash TM parameters inside the test.
In order to achieve this, you'll have to follow these steps:
-
Create custom fields in Squash TM and bind them to the project bearing the test plan to execute.
-
Make sure that the code fields of the parameters correspond to the names of the existing parameters present in the UFT solution.
Below is an example of a UFT solution and the corresponding Squash TM test case automation:
4. Supported versions
Squash AUTOM and Squash DEVOPS have been validated with UFT 15.0.2. Any recent version should work properly.
Automated suite execution
The steps to run an automated test plan in Squash TM are the usual ones:
-
Get to the execution plan of the selected Iteration or Test Suite.
-
Run the test using one of the button on the screen below:
- An Overview of automated test executions popup shows up.
Info
The execution overview popup contains a new section displaying the ongoing executions performed by the Squash Orchestrator. However, the state of the executions are not updated once launched in the current version.
Published results after a Squash TM test plan execution
Independently from the means used to trigger a test plan execution (from Squash TM or a Jenkins pipeline), the kind of results published in Squash TM at the end of the execution of a test plan will differ depending on your using a Squash AUTOM Community or Squash AUTOM Premium licence.
Squash AUTOM Community
After the execution of a Squash TM test plan (iteration or test suite), the following information is updated:
-
ITPIs (Iteration Test Plan Item, see the Squash TM glossary) status update.
-
Automated suite status update.
-
The Allure type report containing all the results from the executed tests.
-
The various ITPIs execution reports are accessible from the Automated Suites tab of the iteration or test suite:
Info
All the results from the automated suite are compiled in an Allure type report, available in the list of reports as a .tar archive.
For more information on the means to exploit and customize the Allure report, please refer to the Allure documentation.
This, however, doesn't happen:
- Creation of a new execution for each executed ITPI.
Squash AUTOM Premium
If you are using the Squash AUTOM Premium components, you have access to two types of results publication:
- Light (default value).
- Full.
The choice of publication type is operated on a project basis by accessing the configuration of the Squash TM Result Publisher plugin from the Plugins tab of your project page, inside the Administration Tab:
Light results publication
By choosing the "Light" results publication, the following information is updated after the execution of a Squash TM test plan (iteration or test suite):
-
ITPIs status update.
-
Automated suite status update.
-
The Allure type report containing all the results from the executed tests.
-
The various ITPIs execution reports are accessible from the Automated Suites tab of the iteration or test suite:
Info
All the results from the automated suite are compiled in an Allure type report, available in the list of reports as a .tar archive.
For more information on the means to exploit and customize the Allure report, please refer to the Allure documentation.
This, however, doesn't happen:
- Creation of a new execution for each executed ITPI.
Full results publication
By choosing the "Full" results publication, the following information is updated after the execution of a Squash TM test plan (iteration or test suite):
-
ITPIs status update.
-
Creation of a new execution for each executed ITPI.
-
Automated suite status update.
-
The Allure type report containing all the results from the executed tests.
-
The execution reports of the various executions can be accessed from the Automated Suites tab of the iteration or test suite, or from the execution page (the reports are present in the attached files):
Info
All the results from the automated suite are compiled in an Allure type report, available in the list of reports as a .tar archive.
For more information on the means to exploit and customize the Allure report, please refer to the Allure documentation.