Automation with JUnit

Test reference in Squash TM

In order to bind a Squash TM test case to a JUnit automated test, the content of the Automated test reference field of the Automation block of a test case must have the following format:

[repository]/[project]/[test_class]#[test_method]
or
[repository]/[test_class]#[test_method]
or
[repository]/[project]/[test_class]
or
[repository]/[test_class]
(The reference contains zero or one # character.)

with:

• [repository]: Name of the Git repository.

• [project]: Path of the directory of the JUnit project (i.e. the directory containing the pom.xml file).
  This parameter is optional, i.e. if the pom.xml file is at the root of the repository, this path is absent.

• [test_class]: Qualified name of the test class.

• [test_method]: Name of the method to test in the test class.
  This parameter is optional, i.e. it may be absent.

Nested tests

In case of @Nested classes, the test reference must have the following format:

[repository]/[project]/[test_class]*
or
[repository]/[test_class]*
or
[repository]/[project]/[test_class]$[nested_class]*
or
[repository]/[test_class]$[nested_class]*
or
[repository]/[test_class]$[nested_class]#[test_method]

There can be a class nested at any level, e.g. [repository]/[test_class]$[nested_class]$[sub_nested_class]$[sub_sub_nested_class]* or [repository]/[test_class]$[nested_class]$[sub_nested_class]#[test_method].

The * wildcard

The * wildcard is necessary to execute tests in all nested classes.
It must not be used if a test method is targeted.

with:

• [repository]: Name of the Git repository.

• [project]: Path of the directory of the JUnit project (i.e. the directory containing the pom.xml file).
  This parameter is optional, i.e. if the pom.xml file is at the root of the repository, this path is absent.

• [test_class]: Qualified name of the test class.

• [nested_class]: Name of the nested class.

• [test_method]: Name of the method to test in the nested class.
  This parameter is optional, i.e. it may be absent.

Surefire and JUnit version

A Surefire version 3.0.0-M6 or later is required to execute nested tests.
@Nested was tested using JUnit version 5.5.2 and later versions, as we encountered problems with earlier versions.

Determination of the result of the test case

If a specific method [test_method] is not specified or if the method is executed several times (e.g. @RepeatedTest), the result of the Squash TM test case is calculated by taking into account the individual results of each test:

• If at least one test has an Error status (in case of a technical issue), the status of the execution will be Blocked.
• If at least one test fails functionally and none of the other ones has an Error status, the status of the execution will be Failed.
• If all tests succeed, the status of the execution will be Success.

Below is an example of a test class and the corresponding Squash TM test case automation:

JUnit 5

JUnit 5 is a very rich test framework and offers a large combination of possible configurations.
Squash Orchestrator uses mvn test -Dtest=testclass#testmethod to run each test.
If you are using the JUnit 5 platform in a simple way, this command line works.

Example 1:
pom.xml file to run tests written in JUnit 5:

<project>
    <modelVersion>4.0.0</modelVersion>
    <groupId>…</groupId>
    <artifactId>…</artifactId>
    <version>…</version>
    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>5.8.2</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>5.8.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.2</version>
            </plugin>
        </plugins>
    </build>
</project>

Example 2:
pom.xml file to run tests written in JUnit 4 (in a JUnit 5 environment):

<project>
    <modelVersion>4.0.0</modelVersion>
    <groupId>…</groupId>
    <artifactId>…</artifactId>
    <version>…</version>
    <dependencies>
        <dependency>
            <groupId>org.junit.vintage</groupId>
            <artifactId>junit-vintage-engine</artifactId>
            <version>5.8.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

If you are using a more complex configuration, for example with junit-platform-surefire-provider or with junit-platform-runner, please check that mvn test -Dtest=testclass#testmethod works.

Supported types of JUnit 5 tests

The following test types are supported:

• @Test
• @RepeatedTest
• @ParameterizedTest
• @TestFactory
• @TestTemplate
• @Nested classes

The support of @DisplayName custom display names is still unavailable.

3.0.0-M4/3.0.0-M5 Surefire bug

The versions 3.0.0-M4 and 3.0.0-M5 Surefire versions contain a bug (SUREFIRE-1857) resulting in the Surefire report not containing failure messages. Because of this, the generated Allure reports will also not contain the failure messages. Hence, it is advised to not use these Surefire releases.

Nature of the exploitable Squash TM parameters

The exploitable Squash TM parameters will differ depending on whether you're using the Community/Premium or Ultimate version of Squash.

Here is a table showing the exploitable parameters (these parameters are transmitted as test parameters, see below, Squash TM does not generate global parameters):

Nature	Key	Community/Premium	Ultimate
Name of the dataset	DSNAME	✅	✅
Dataset parameter	DS_[name]	✅	✅
Test case reference	TC_REFERENCE	✅	✅
Test case internal UUID	TC_UUID	✅	✅
Test case custom field	TC_CUF_[code]	✅	✅
Iteration custom field	IT_CUF_[code]	❌	✅
Campaign custom field	CPG_CUF_[code]	❌	✅
Test suite custom field	TS_CUF_[code]	❌	✅

Legend:

• [code]: Value of the "Code" of a custom field
• [name]: Parameter name as filled in Squash TM

As indicated, Squash TM adds a prefix to the code of the transmitted custom field. Make sure to take it into account.

Refer to the Squash TM documentation for more information about custom fields.

Parameters usage

It is possible, when running JUnit tests, to exploit parameters within it. A parameter can be a test parameter or a global parameter. Squash TM transmits only test parameters. Test parameters and global parameters can be used in the case of a launch from a CI/CD pipeline with the junit/params action.

To do this, the following steps must be followed:

• Import opentestfactory-java-param-library into the Maven project containing the tests to run by adding to the pom.xml file:

  • the following Maven repository:

  <repositories>
      <repository>
          <id>org.squashtest.repo.release</id>
          <name>Squashtest repository for releases</name>
          <url>https://nexus.squashtest.org/nexus/repository/maven-squashtest-public-releases/</url>
      </repository>
  </repositories>
  

  • the following Maven dependency:

  <dependencies>
      <dependency>
          <groupId>org.opentestfactory.util</groupId>
          <artifactId>opentestfactory-java-param-library</artifactId>
          <version>1.1.0</version>
      </dependency>
  </dependencies>
  

• You can then retrieve the value of a parameter of the desired type using the following syntax:

  ParameterService.INSTANCE.getString("paramName");
  ParameterService.INSTANCE.getInt("paramName");
  ParameterService.INSTANCE.getFloat("paramName");
  ParameterService.INSTANCE.getDouble("paramName");
  ParameterService.INSTANCE.getBoolean("paramName");
  

  The above methods look for the desired parameter in the test parameters; if they cannot find it, they then look for it in the global parameters.
  These methods throw a ParameterNotFoundException if the parameter is not found. If the parameter is found but cannot be converted to the desired type, a ParameterFormatException is thrown. Consider handling these exceptions in your test classes.
  Warning: the conversion of Boolean data returns true when the character string to be converted is equal to "true" (whatever the case), false in all other cases; but never propagates an exception.

• It is also possible to define a default value in the case where the parameter does not exist by using the following syntax:

  ParameterService.INSTANCE.getString("paramName", defaultValue);
  ParameterService.INSTANCE.getInt("paramName", defaultValue);
  ParameterService.INSTANCE.getFloat("paramName", defaultValue);
  ParameterService.INSTANCE.getDouble("paramName", defaultValue);
  ParameterService.INSTANCE.getBoolean("paramName", defaultValue);
  

  The above methods therefore do not propagate a ParameterNotFoundException when the parameter sought is not found but propagate a ParameterFormatException if the parameter is found, but cannot be converted to the desired type.

• It is also possible to target a test parameter or a global parameter with specific methods. As with the previous methods, they are available in versions with and without default values. Here are a few examples:

  ParameterService.INSTANCE.getTestString("paramName");
  ParameterService.INSTANCE.getGlobalInt("paramName");
  ParameterService.INSTANCE.getTestFloat("paramName", defaultValue);
  ParameterService.INSTANCE.getGlobalBoolean("paramName", defaultValue);
  

Example

Below is an example of a JUnit test file and the automation of the associated Squash TM test case:

package squash.tfauto;

import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.opentestfactory.exception.ParameterException;
import org.opentestfactory.exception.ParameterFormatException;
import org.opentestfactory.util.ParameterService;

public class ParameterTest {

  @Test
  public void sub() throws ParameterException {
    int var1 = ParameterService.INSTANCE.getInt("DS_VAR1");
    int var2 = ParameterService.INSTANCE.getTestInt("DS_VAR2");
    int res = ParameterService.INSTANCE.getTestInt("DS_RES");
    Assertions.assertEquals(res, var1 - var2, "Wrong calculation result");
  }

  @Test
  public void add() throws ParameterFormatException {
    float var1 = ParameterService.INSTANCE.getFloat("DS_FAKE_VAR1", 42.21f);
    float var2 = ParameterService.INSTANCE.getTestFloat("DS_FAKE_VAR2", 15.23f);
    float res = ParameterService.INSTANCE.getTestFloat("DS_FAKE_RES", 57.44f);
    Assertions.assertEquals(res, var1 + var2, "Wrong calculation result");
  }

  @Test
  public void displayLabel() throws ParameterException {
    Boolean isDisplay = ParameterService.INSTANCE.getBoolean("DS_DISPLAY");
    String label = "default";
    if (isDisplay) {
      label = ParameterService.INSTANCE.getString("TC_CUF_LABEL");
    }
    Assertions.assertEquals("Squash", label, "Wrong label displayed");
  }
}

Adding parameters to the test launch command line

You can pass additional parameters to the mvn test command using the JUNIT_EXTRA_OPTIONS environment variable. Here is an example of how to define an environment variable in Squash TM and associate it with the orchestrator.

Some parameters are already defined in the mvn test command used to launch tests:

mvn test \
  -f "{POM_path}" -Dtest={test_name} \
  -Dmaven.test.failure.ignore=true -DfailIfNoTests=true \
  --log-file {junit_run_log_path} $JUNIT_EXTRA_OPTIONS

You must avoid passing, via the JUNIT_EXTRA_OPTIONS variable, the command line parameters that conflict with the parameters already used, or the parameters that impact the generation or alter the path of the reports expected by the orchestrator (view the report list).

Non-support of the space character on Linux

Squash Orchestrator currently does not support the space character () in the JUNIT_EXTRA_OPTIONS environment variable for tests executed in a Linux execution environment.

Performance boost

You can use Maven Daemon, a Maven wrapper, to speed up the JUnit test executions: see the technical details chapter. It is useful when there are many short tests to execute. If each test takes over several minutes to run, Maven Daemon is of no real interest.

Supported versions

Squash has been validated with

• JUnit 4.12, any JUnit 4 version more recent should work properly
• JUnit 5.3.2, any JUnit 5 version should work properly

Additionally, it is advised to use Surefire 2.19.1 or later (but not 3.0.0-M4/3.0.0-M5 as described above).