UI variables take precedence over build.env

Migrated issue

Reported by: jonathan1055

Related to !299 (merged)

Problem/Motivation

When running a pipeline from the UI, the CSpell job fails with a curl error, due to _CURL_TEMPLATES_REPO and _CURL_TEMPLATES_REF being empty.

See slack discussion https://app.slack.com/client/T06GX3JTS/CGKLP028K

This likely to be caused by the changes in #3487169: Set IMAGE and CURL variables for all subsequent jobs. The variables from the UI may take precedence over variables in build.env. See https://docs.gitlab.com/ee/ci/variables/index.html#cicd-variable-precedence. UI would be 3 and build.env would be 7.

Proposed resolution

Consider/test if we can do

_CURL_TEMPLATES_REPO: $_GITLAB_TEMPLATES_REPO
_CURL_TEMPLATES_REF: $_GITLAB_TEMPLATES_REF

or

Move the _CURL_* variables to the hidden file so they are not available via the UI

or

Delete the _CURL_* variables from the file and hence they will no show in the UI. They are only needed in .gitlab-ci.yml

or

otherwise we might need to call calculate-gitlab-ref again in all the jobs that need it.

Remaining tasks

User interface changes

API changes

Data model changes