Squash TM test execution plan retrieval with a workflow

Squash gives you the possibility to insert, in a CI/CD pipeline, a step to retrieve and execute an automated test plan defined in Squash TM using a workflow written in YAML or JSON. In the case of Jenkins, the configuration of this step can be simplified with a plugin, see the corresponding page of this guide.
You can find more information regarding the redaction of a workflow in the OpenTestFactory Orchestrator documentation.

Focus

For this functionality to work properly, Test Plan Retriever Plugin and Result Publisher Plugin must be installed on the targeted Squash TM instance.

Prerequisites

A user belonging to the Test automation server group must exist in Squash TM.

Integration of the Squash TM execution plan retrieval step into a workflow

The execution plan, iteration or test suite, must contain at least one ITPI (Iteration Test Plan Item, see the Squash TM glossary) linked to an automated test case, as described in the instructions here.

In order to retrieve this execution plan from Squash TM with a workflow, you need to call the corresponding generator action.

Here is a simple example of such a workflow in JSON format:

{
    "apiVersion": "opentestfactory.org/v1alpha1",
    "kind": "Workflow",
    "metadata": {
        "name": "Simple Workflow"
    },
    "defaults": {
        "runs-on":"ssh"
    },
    "jobs": {
        "explicitJob": {
            "runs-on":"ssh",
            "generator":"tm.squashtest.org/tm.generator@v1",
            "with": {
                "testPlanUuid":"1e2ae123-6b67-44b2-b229-274ea17ad489",
                "testPlanType":"Iteration",
                "squashTMUrl":"https://squashtm.example.com/squash",
                "squashTMAutomatedServerLogin":"tfserver",
                "squashTMAutomatedServerPassword":"tfserver",
                "technologyLabels":{
                    "ranorex": ["windows","ranorex"],
                    "robotframework": ["linux","robotframework"],
                    "junit": ["linux","junit"]
                }
            }
        }
    }
}

A Squash TM generator step must contain the following parameters:

• testPlanType: Defines the type of test plan to retrieve in Squash TM. Only the values Iteration and TestSuite are accepted.

• testPlanUuid: This is the UUID (Universally Unique IDentifier, see the Squash TM documentation) of the requested test plan. It can be found in the Description block by clicking on the Information tab of the iteration or test suite in Squash TM.

• squashTMUrl: URL of the targeted Squash TM.

• squashTMAutomatedServerLogin: Name of the Test automation server group user to log into Squash TM.

• squashTMAutomatedServerPassword: Password of the Test automation server group user to log into Squash TM.

Optional fields:

• tagLabel: Specific to the Ultimate version - It refers to the name of the tag type custom field on which the test cases to retrieve are to be filtered.
  It is not possible to specify more than one.

• tagValue: Specific to the Ultimate version - It refers to the value of the tag type custom field on which the test cases to retrieve are to be filtered.
  It is possible to specify multiple ones separated by | (example value1|value2). There has to be at least one value specified for the test case to be taken into account.

Focus

If one of the two tagLabel or tagValue fields is present, the other one must also be specified.

• technologyLabels: To use if you retrieve an execution plan containing tests which must be executed on different environments.
  It enables you to specify, for each automation framework, targeted environment execution's tags.
  They will be added to the ones specified in the job's runs-on attribute.
  For a technology, tags are defined by adding an entry to the field list with format "technology prefix": ["tag1", "tag2"].

Info

By default, jobs created by the Generator after retrieving an execution plan are tagged with the corresponding technology prefix (technology name in lower case with no space, for example "robotframework" for Robot Framework) in addition of the ones specified in Generator job's runs-on attribute.

Execution order of tests

The only order assured by Squash is that, for a given Git repository in a given iteration/test suite, the tests will be executed in the order defined in Squash TM.
If a test iteration/suite contains automated tests belonging to multiple Git repositories, the order of execution of tests from one repository relative to the execution of tests from another repository is undefined.
If several test iterations/suites are launched from Squash TM (resp. the CI/CD pipeline), the order of execution of the tests of one iteration/suite relative to the execution of the tests of another iteration/suite is undefined. (Except in the trivial case where a suitable execution environment is available and the first suite/iteration was able to start before the Squash TM user – resp. the CI/CD pipeline – launches another one.)

Usage of the orchestrator with inception execution environment

OpenTestFactory, and therefore Squash, have a special execution environment called inception to use an alternative use mode for generator type jobs.

What is the purpose of inception environment?

Usual execution process of a generator job is to retrieve an execution plan from Squash TM and then execute automated tests of this plan on dedicated execution environment. When an inception execution environment is used by a generator job, there is no test execution following execution plan retrieval. Instead, result publication step to Squash TM is run based on a set of reports sent upstream to the orchestrator.

How to use inception environment in a workflow

Below is a workflow example in YAML format using inception execution environment and retrieving a Squash TM execution plan.

metadata:
  name: Test Inception
resources:
  files:
  - report1
  - report2
  - report3
jobs:
  prepare:
    runs-on: inception
    steps:
    - uses: actions/prepare-inception@v1
      with:
        report.html: ${{ resources.files.report1 }}
        output.xml: ${{ resources.files.report2 }}
        log.html: ${{ resources.files.report3 }}
  robot:
    runs-on: inception
    needs: [prepare]
    generator: tm.squashtest.org/tm.generator@v1
    with:
      squashTMUrl: https://squashtm.example.com/squash
      squashTMAutomatedServerLogin: ${{ variables.SQUASH_USER }}
      squashTMAutomatedServerPassword: ${{ variables.SQUASH_PASSWORD }}
      testPlanUuid: 1e2ae123-6b67-44b2-b229-274ea17ad489
      testPlanType: Iteration

There are several differences compared to the workflow of the previous section:

• The files section in the resources part: it lists the reports provided to the orchestrator. Information published to Squash TM will be extracted from these reports.

• A preparation job, called prepare in this example: it uses actions/prepare-inception@v1 action provider and allows to indicate what reports, usually generated by the test execution, match with the reports indicated in the files section.
  The list of reports supported by Squash Orchestrator is indicated on the page corresponding to each test technology:

  • Agilitest💎
  • Cucumber JVM
  • Cypress
  • JUnit
  • Katalon💎
  • Playwright
  • Postman
  • Ranorex💎
  • Robot Framework
  • SKF
  • SoapUI
  • UFT💎💎

  The first report in the list (which is the Surefire report for most test technologies) must be present for the orchestrator to determine the test result.

• The generator job: it is very similar to the one of the previous section with two differences:

  • It uses inception environment in the runs-on field
  • It specifies that the preparation job is a prerequisite for its execution with the line needs: [prepare]

Such a workflow can be run with the following command:

BashCMDPowerShell

curl -X POST \
     -H "Authorization: Bearer ${TOKEN}" \
     -F workflow=@Squashfile \
     -F report1=@report1.html \
     -F report2=@report2.xml \
     -F report3=@report3.xml \
     -F variables="SQUASH_USER=${USER}\nSQUASH_PASSWORD=${PASSWD}" \
     http://orchestrator.example.com/workflows

curl -X POST ^
     -H "Authorization: Bearer %TOKEN%" ^
     -F workflow=@Squashfile ^
     -F report1=@report1.html ^
     -F report2=@report2.xml ^
     -F report3=@report3.xml ^
     -F variables="SQUASH_USER=%USER%\nSQUASH_PASSWORD=%PASSWD%" ^
     http://orchestrator.example.com/workflows

curl -X POST `
     -H "Authorization: Bearer $Env:TOKEN" `
     -F workflow=@Squashfile `
     -F report1=@report1.html `
     -F report2=@report2.xml `
     -F report3=@report3.xml `
     -F variables="SQUASH_USER=$Env:USER\nSQUASH_PASSWORD=$Env:PASSWD" `
     http://orchestrator.example.com/workflows

Note in this command the parameters specifying the reports to be sent by the orchestrator to Squash TM.

A workflow using inception execution environment can also be run from a Jenkins pipeline thanks to the Jenkins plugin step.

Squash TM parameters to exploit in an automated test

By executing a workflow retrieving a Squash TM execution plan, Squash TM passes various pieces of information on execution plan items that can be exploited during test case execution.

The following frameworks can exploit Squash TM parameters during test execution:

• Cucumber JVM
• Agilitest💎
• Cypress
• JUnit
• Katalon💎
• Playwright
• Postman
• Ranorex💎
• Robot Framework
• SKF
• UFT💎

Each of these pages describes the parameters that can be retrieved by the test script and how to retrieve them.

Results publication in Squash TM at the end of the execution

The results and reports of the executed tests are transmitted and accessible in Squash TM in the same manner as for an execution initiated from it, refer to this paragraph.