Add a variable for run-tests.sh concurrency, default stays at 32
Problem/Motivation
Companion issue to #3450616: Optimize test order when --directory is used for core.
The default run-tests command in the gitlab CI templates runs tests with --concurrency 32. This is quite a bit higher than is likely to be the optimum with KUBERNETES_CPU_REQUEST: 2.
I think we should either:
1. Add a variable for concurrency and set it to something a bit above, but not dramatically above, the cpu request variable - maybe 8? Then projects can customize it from there.
2. Do #1 but instead of defaulting it to 8 do something like max(32, CPUS * 2) to try to intelligently set it based on the CPU request.
The bigger problem here is that functional javascript tests need more CPU allocation because chrome adds extra CPU needs, functional and kernel tests are generally a bit more i/o heavy ( and don't need chrome) so might be OK with less CPU allocation, but kernel and unit tests are CPU heavy again because there's very little if any i/o.
Given all test types are run together, and all contrib modules have different combinations of tests, picking a good number is going to be hard, but I have a pretty good feeling it shouldn't be 32.
Once #3450616: Optimize test order when --directory is used is in core, the slowest tests will run first, with unit tests last - this should max out concurrency for the duration of the test run, meaning more tests can finish quicker even with less concurrency (because functional tests will finish at different times, and while the last one is finishing, remaining threads can pick up and finish all the kernel and unit tests very quickly, instead of doing the fast test firsts, and only then starting on the slowest tests).
Steps to reproduce
Proposed resolution
Remaining tasks
User interface changes
API changes
Data model changes
Related issue: Issue #3450616