[PP-1] Add tests for variable combinations of OPT_IN_* and SKIP_*

Migrated issue

Reported by: jurgenhaas

Related to !114

Problem/Motivation

In #3414391: Define extra rule to jobs that have "needs: composer" we've introduced consistent behaviour for all the possible combinations of OPT_IN_* and SKIP_* variables in GitLab CI configurations of a project.

It is very likely that even more combinations will be introduced as new features get added to the templates.

This makes it impossible (if it isn't already) to test all those combinations and make sure that they ain't broken by any of the changes to the template files.

Proposed resolution

We can fix that by adding all the tests to the CI of this template project. Here is how that could look like:

  • We build a reference project that contains several branches, each of which with files present or absent, which are relevant for certain jobs to be run or not, e.g. *.js files for ESLINT or mkdocs.yml for PAGES
  • The CI configuration of that reference project gets a job in the .pre stage which can only be triggered manually. That will allow us to create pipelines that can be analysed but actually won't be executed - to save resources
  • Then we define a matrix of variable combinations that we want to test, together with a list of expected jobs, that should be created in the respective pipeline
  • The test job will then loop through that list and create a pipeline for each, together with the necessary variables. We can use the glab CLI tool for this and that will create pipelines, but they won't execute
  • With the same tool, we can receive the pipeline details which should contain a list of contained jobs. This can be compared to the list of expected jobs, and if that doesn't match, the pipeline fails and we get notified that something is broken

This approach is fully automated and easy to extend, since only the matrix needs to be extended when new features get added to the templates.

It should be considered whether that test should be executed by default or only under certain circumstances.

Related issue: Issue #3414391

Related issue: Issue #3447071

Edited by drupalbot