CI: introduce a nightly-only `slow-motion cypress E2E` CI job to auto-detect response latency-sensitive tests
>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3503377. --> Reported by: [wim leers](https://www.drupal.org/user/99777) Related to !594 >>> <h3 id="overview">Overview</h3> <p>Ever since <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-7"><a href="https://www.drupal.org/project/experience_builder/issues/3499931" title="Status: Closed (fixed)">#3499931: HTTP API for code component config entities</a></span> landed, the <code>drag-and-drop</code> and <code>publish-button</code> e2e tests are failing almost consistently.</p> <p>@f.mazeikis determined the root cause: a significant increase in OpenAPI validation duration due to it references JSON Schema.</p> <p>(E2E tests <em>do</em> run OpenAPI validation of all requests/responses, as always was the intent, ever since #3500110 &mdash; see <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-7"><a href="https://www.drupal.org/project/experience_builder/issues/3500110" title="Status: Closed (fixed)">#3500110: CI: PHP asserts not running in CI/during tests due to core regression</a></span> for proof.)</p> <p><code>/xb/api/config/component</code> takes:</p> <ul> <li>~1.5&ndash;1.8s on my system with HEAD</li> <li>~0.6 s on my system with<br> <pre>diff --git a/openapi.yml b/openapi.yml<br>index a36d1325..c7eeb1de 100644<br>--- a/openapi.yml<br>+++ b/openapi.yml<br>@@ -798,7 +798,7 @@ components:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; items:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; type: string<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; props:<br>-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $ref: 'https://json-schema.org/draft-04/schema'<br>+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; type: object<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; slots:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; type: array<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; items:</pre></li> </ul> <h3 id="proposed-resolution">Proposed resolution</h3> <p>Temporarily make the OpenAPI validation less thorough.</p> <p>Follow-up to improve our CI:</p> <ol> <li> <ul> <li>First run Cypress E2E tests with assertions enabled aka with OpenAPI testing </li><li>If that run fails, do a second run without, which should be faster. </li></ul> </li> <li>Inspired by @jessebaker: we should probably do a secondary Cypress E2E test run during nightly test runs only (because it&rsquo;d be super slow): <strong>every request should be <code>cy.intercept()</code>ed and delayed by 1 second. If that fails, but the non-artificially-slowed-down test passes, then we know our E2E tests are not sufficiently defensive! &#128516; </strong></li> </ol> <p>That way, we keep the benefits of OpenAPI validation, but are not held hostage by it. CI runs become faster with two things:</p> <ul> <li>more defensive E2E tests </li><li>faster OpenAPI validation </li></ul> <h3 id="ui-changes">User interface changes</h3> <p>None.</p> > Related issue: [Issue #3500110](https://www.drupal.org/node/3500110) > Related issue: [Issue #3499931](https://www.drupal.org/node/3499931)
issue