Quality gate

The quality gate is one of the Squash Orchestrator components. It may be used to fine-tune the conditions for success or failure of a pipeline that contains a test execution stage. The quality gate makes it possible to define evaluation rules that specify an expected success threshold and a test evaluation scope. For instance, the user can cause the pipeline to fail at the test stage if less than 90% of tests of very high importance are successful. To do this, he must provide a definition file to the quality gate service.

Quality gate definition file

This file must be in YAML format (.yaml or .yml) and define at least one quality gate containing at least one rule. Here is an example of a basic definition:

qualitygates:
 - name: my.quality.gate
   rules:
   - name: JUnit tests
     rule:
       scope: (test.technology=='junit') && (test.importance=='VERY_HIGH')
       threshold: 95%
       failure-status: [failure]

In this example, only one quality gate my.quality.gate is specified. It contains one rule, JUnit tests, which applies to the JUnit tests of very high importance. The quality gate is passed if at least 95% of tests are evaluated as successful.

Only .rules.name and .rule.failure-status properties are optional. If a rule has no name, an UUID will be provided instead. If failure-status is omitted, it will default to ['failure', 'error', 'blocked'].

A definition file may define several quality gates and a quality gate may contain as many rules as needed.

Threshold and scope definition

The rule threshold is a percentage between 0% and 100%.

The rule scope makes use of expressions and functions that are described in the OpenTestFactory documentation.

The quality gate uses the orchestrator test context. Its properties are detailed in the table below. In this context, users can also refer to test plan (iteration or test suite) properties, using test.collection property, as well as access datasets or CUFs through test.data.

Property availability

All of these priorities are available for Squash TM 6.0 or later.
For older versions, only test.technology, test.uses, test.runs-on, and test.job are usable.

Legend

️‡ indicates a property or a value available with the Premium components. An overview of the Premium features is available here. To benefit from these or to ask for more information, check our website or contact us.

Property	Type	Description	Values
`test`
`test.technology`	string	Test technology.	`cucumber, cucumber5, cypress, junit, postman, robotframework, skf, soapui, agilitest‡, katalon‡, ranorex‡, uft‡`
`test.uses`	string	Action used to execute the test.	See providers actions in the OpenTestFactory documentation.
`test.runs-on`	object	Execution environment tags.	Use `contains()` function to apply filtering.
`test.managed`	boolean	Indicator of a test managed by a test referential	`true` if and only if the test is managed by Squash TM (i.e. it is launched during the execution of a Squash TM test case to which it is linked).
`test.job`	string	Name of the job running the test execution.
`test.name`	string	Test case name in Squash TM.	See the respective field in the "Test Cases" workspace.
`test.technology-name`	string	Automated test technology, as displayed in Squash TM.	See the "Automated test technology" dropdown list in the test case "Automation" block.
`test.reference`	string	Automated test case reference in Squash TM.	See the "Automated test reference" field in the test case "Automation" block.
`test.importance`	string	Test case importance in Squash TM.	`VERY_HIGH, HIGH, MEDIUM, LOW`
`test.nature`	string	Test case nature in Squash TM.	`NAT_UNDEFINED, NAT_FUNCTIONAL_TESTING, NAT_BUSINESS_TESTING, NAT_USER_TESTING, NAT_NON_FUNCTIONAL_TESTING, NAT_PERFORMANCE_TESTING, NAT_SECURITY_TESTING, NAT_ATDD`
`test.type`	string	Test case type in Squash TM.	`TYP_UNDEFINED, TYP_COMPLIANCE_TESTING, TYP_CORRECTION_TESTING, TYP_REGRESSION_TESTING, TYP_EVOLUTION_TESTING, TYP_END_TO_END_TESTING, TYP_PARTNER_TESTING`
`test.path`	object	Test case path in Squash TM ("Test Cases" workspace).	Use `contains()` function to apply filtering.
`test.collection`
`test.collection.path`	object	Test plan (iteration or test suite) path in Squash TM ("Campaigns" workspace).	Use `contains()` function to apply filtering.
`test.collection.type`	string	Test plan type (iteration or test suite) in Squash TM.	`iteration`, `test suite`
`test.collection.uuid`	string	Test plan UUID in Squash TM.	See the "UUID" field of the test iteration or the test suite "Information" block.
`test.data`
`test.data.DSNAME`	string	Dataset name in Squash TM.	See the "NAME" field of the test case "Parameters and Datasets" block.
`test.data.DS_{param_name}`	string	Parameter name in Squash TM.	As defined in the test case "Parameters and Datasets" block.
`test.data.TC_REFERENCE`	string	Test case référence in Squash TM.	See the "Test case reference" field above the test case "Information" block.
`test.data.TC_UUID`	string	Test case UUID.
`test.data.TC_CUF_{CUF_CODE}`	string	Test case custom user field in Squash TM.
`test.data.TS_CUF_{CUF_CODE}`‡	string	Test suite custom user field in Squash TM.
`test.data.IT_CUF_{CUF_CODE}`‡	string	Test iteration custom user field in Squash TM.
`test.data.CPG_CUF_{CUF_CODE}`‡	string	Test campaign custom user field in Squash TM.

Apply the quality gate to a workflow

Provide the definition file to the service

To apply a quality gate to a workflow, the user must first provide the definition file to the service. There are two ways to do it:

1) Load the file when launching the orchestrator. In this case, just add two parameters to the run command:

the definition file mounting point on the orchestrator's docker image;
the QUALITYGATE_DEFINITIONS environment variable, that must contain the definition file path on this image.

BashCMDPowerShell

docker run ... \
           -v /path/to/qg_def/my_qualitygate.yaml:/app/qualitygate/my_qualitygate.yaml \
           -e QUALITYGATE_DEFINITIONS=/app/qualitygate/my_qualitygate.yaml \
           ...

docker run ... ^
           -v d:\path\to\qg_def\my_qualitygate.yaml:/app/qualitygate/my_qualitygate.yaml ^
           -e QUALITYGATE_DEFINITIONS=/app/qualitygate/my_qualitygate.yaml ^
           ...

docker run ... `
           -v d:\path\to\qg_def\my_qualitygate.yaml:/app/qualitygate/my_qualitygate.yaml `
           -e QUALITYGATE_DEFINITIONS=/app/qualitygate/my_qualitygate.yaml `
           ...

Two quality gates are always provided when the service is launched. The strict quality gate evaluates all executed tests with a 100% threshold and default failure-status values. The passing quality gate has 0% threshold and will be successful regardless of test execution results.

2) Specify the definition file via the opentf-ctl get qualitygate command --using option (detailed below).

Warning

If the definition file is provided using the get qualitygate command, only the quality gates defined in this file can be used. The service-level quality gates become unavailable.

Evaluate the execution result

To evaluate a workflow execution result, you need to use the opentf-ctl get qualitygate command from the orchestrator tools.

The opentf-ctl get qualitygate {workflow_id} {options} command evaluates a workflow using all the rules of the quality gate specified by the user. This command returns the evaluation result for each rule and the general workflow evaluation result:

opentf-ctl get qualitygate a13f0572-b23b-40bc-a6eb-a12429f0143c --mode my.quality.gate

RULE,RESULT,TESTS_IN_SCOPE,TESTS_FAILED,TESTS_PASSED,SUCCESS_RATIO
JUnit tests,FAILURE,50,10,40,80.0%
Workflow a13f0572-b23b-40bc-a6eb-a12429f0143c failed the quality gate using mode my.quality.gate.

{workflow_id} or the evaluated workflow UUID is the mandatory parameter for the command.

The command has the following options:

--mode or -m. The quality gate name.
--using or -u. The user-provided definition file path.
--output wide or -o wide. Adds two columns to the command output: THRESHOLD and SCOPE.

Info

If you want to specify the columns displayed in the command output, you may use the --output custom-columns option: see the OpenTestFactory documentation.

When the definition file is provided via the --using option, the --mode option is mandatory. If the definition file is loaded at the orchestrator level and the --mode option is not specified, the quality gate strict will be applied by default.

Each rule returns FAILURE, SUCCESS or NOTEST (if no test matching rule scope has been found). The general result is FAILURE ("workflow failed the quality gate") if at least one rule has failed, and SUCCESS ("workflow passed the quality gate") if all the rules return SUCCESS. If all the rules return NOTEST, the general result is also NOTEST ("workflow contains no test matching quality gate scopes").

The get qualitygate command return codes are:

0 if the workflow passed the quality gate or the general result is NOTEST;
101 if the specified workflow is still running;
102 if the quality gate failed for the specified workflow.

Examples

Apply a user-defined quality gate

Let's take a pipeline that executes a Squash TM iteration containing Selenium and Robot Framework tests. We want the pipeline to succeed in the following case:

at least 90% of Selenium-tagged tests of very high and high importance are successful and
at least 75% of Robot Framework tests that do not use API tests dataset are successful.

This quality gate is not specified in the orchestrator-level quality gate definition file.

In this scenario, the user must first create his own definition file:

qualitygates:
 - name: custom.quality.gate
   rules:
   - name: Selenium tests
     rule:
       scope: (test.data.TC_CUF_TAG=='Selenium') && ((test.importance=='VERY_HIGH') || (test.importance=='HIGH'))
       threshold: 90%
       failure-status: [failure]
   - name: RobotFramework
     rule:
       scope: (test.technology=='robotframework') && (test.data.DSNAME!='API tests')
       threshold: 75%
       failure-status: [failure]

Information

Instead of ((test.importance=='VERY_HIGH') || (test.importance=='HIGH')) condition, you may also use contains(fromJSON('["HIGH", "VERY_HIGH"]'), test.importance).

Assuming the user saves the file as custom_quality_gate.yaml, he must then pass its path to the opentf-ctl get qualitygate command:

opentf-ctl get qualitygate {workflow_id} --mode custom.quality.gate --using /home/user/custom_quality_gate.yaml --output wide

In this example, the --output wide option is used and the command output includes not only the execution information but also the user-defined threshold and scope:

RULE,RESULT,TESTS_IN_SCOPE,TESTS_FAILED,TESTS_PASSED,SUCCESS_RATIO,THRESHOLD,SCOPE
Selenium tests,SUCCESS,20,2,18,90.0%,90%,(test.data.TC_CUF_TAG=='Selenium') && ((test.importance=='VERY_HIGH') || (test.importance=='HIGH'))
RobotFramework,SUCCESS,40,5,35,87.5%,75%,(test.technology=='robotframework') && (test.data.DSNAME!='API tests')
Workflow {workflow_id} passed the quality gate using mode custom.quality.gate.

Apply a rule to all the workflow tests

To apply a rule to all the tests executed in a workflow, just set the scope value to true:

qualitygates:
 - name: my.quality.gate
   rules:
   - name: All tests
     rule:
       scope: 'true'
       threshold: 85%

Using paths in the rule scope

The quality gate allows the use of Squash TM iterations and/or test suite paths, as well as test suite paths, within the rule scope. The function contains(search, item) must be applied in this case, search standing for the property and item for the respective path element name.

The following rule evaluates all the test cases in TNR_main and TNR_smoketest folders of the WebApp project in the Squash TM "Test Cases" workspace:

qualitygates:
 - name: my.quality.gate
   rules:
   - name: TNR
     rule:
       scope: (contains(test.path, 'TNR_main') || contains(test.path, 'TNR_smoketest')) && contains(test.path('WebApp')) 
       threshold: 95%

To add an iteration and/or test suite path in the rule scope, the function contains() must be used with the test.collection.path property. For instance, the following rule will be applied to all the Dropdowns iteration test cases:

qualitygates:
 - name: my.quality.gate
   rules:
   - name: Dropdowns
     rule:
       scope: (test.collection.type=='iteration') && contains(test.collection.path, 'Dropdowns')
       threshold: 100%

Using the quality gate in a Jenkins pipeline

It is possible to apply the quality gate in a Jenkins pipeline using the opentf-ctl get qualitygate command. The user must have access to a Jenkins instance that is linked to the orchestrator by the opentestfactory-orchestrator-2.1.0.hpi plugin. He must also make sure that the orchestrator tools (opentf-tools) are available in the Jenkins execution environment.

Info

See respective chapters of this documentation for plugin installation, plugin configuration and calling the orchestrator from Jenkins.

When the communication between Jenkins and the orchestrator has been established, call the quality gate from the pipeline using the completed workflow UUID.

Here is an example of a pipeline that launches squash_sample_tests.yaml workflow and applies the orchestrator-level quality gate sample.quality.gate to the completed workflow:

pipeline {
  agent any
  environment {
    WORKFLOW_ID = ''
  }
  stages {
    stage('Run OTF workflow'){
      steps {
        script {
          WORKFLOW_ID = runOTFWorkflow(
            workflowPathName: 'squash_sample_tests.yaml',
            workflowTimeout: '300S',
            serverName:'Orchestrator',
            jobDepth: 2,
            stepDepth: 3,
            dumpOnError: true
          ) 
        }
      }
    }
    stage('Apply quality gate'){
      steps {
        script {
          def qg_command = "opentf-ctl get qualitygate $WORKFLOW_ID --mode sample.quality.gate"
          def qg_result = sh(returnStdout: true, script: qg_command)
          echo "$qg_result"
        }
      }
    }
  }
}

This pipeline uses the WORKFLOW_ID environment variable that can be passed between the pipeline stages. Its value (completed workflow UUID) is retrieved from the runOTFWorkflow() function and passed to the get qualitygate command. The command output is then passed to another variable (qg_result) and displayed in the console.

If the quality gate fails, a 102 error code is returned the pipeline fails. If the workflow passes the quality gate or there are no tests matching evaluation scopes, the command returns 0 and the pipeline successfully completes (or continues if there are other stages).

The user can choose not to stop the pipeline even if the workflow fails the quality gate. In this case, the shell script must be executed with the returnStatus: true option instead of the returnStdout: true option. The pipeline step will return the script return code, which can be handled in the following steps or stages.

Using the quality gate in a GitLab pipeline

It is also possible (and even recommended) to apply the quality gate to a workflow executed in a GitLab pipeline. A complete guide on the Squash Orchestrator integration with the GitLab CI is available in the OpenTestFactory documentation.

Here is an example of a .gitlab-ci.yml file that runs the my_workflow.yaml workflow and applies the quality gate from a user-provided definition file each time the user makes changes to the project:

default:
  image: python:3.8

stages:
  - test

opentf-workflow:
  stage: test
  script:
    - pip install opentf-tools
    - RESULT=$(opentf-ctl run workflow .opentf/workflows/my_workflow.yaml --watch)
    - WORKFLOW_ID=$(echo $RESULT | head -n 1 |awk -F ' ' '{print $2}')
    - opentf-ctl get qualitygate $WORKFLOW_ID --mode sample.quality.gate --using .opentf/qualitygates/my_quality_gate.yaml

The pip install command checks that the most recent version of the orchestrator tools is available on the runner, and the opentf-ctl run... command runs the workflow with the --watch option to follow the execution. The UUID of the workflow is retrieved from the command output to be stored in the WORKFLOW_ID variable.

Finally, the quality gate sample.quality.gate, which is defined in the my_quality_gate.yaml definition file, is applied to the workflow and the get qualitygate command output is displayed. If the workflow fails the quality gate, it will return a 102 error code and the pipeline will fail. Without the quality gate, the pipeline will always be in success regardless of the test results.