CI: Playwright ARIA snapshot testing is highly misleading — subsets are considered valid
>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3543779. --> Reported by: [wim leers](https://www.drupal.org/user/99777) >>> <h3 id="overview">Overview</h3> <p>Over at <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-7"><a href="https://www.drupal.org/project/experience_builder/issues/3543770" title="Status: Closed (fixed)">#3543770: `noUi: true` SDCs do still show up in the UI</a></span>, I pushed a commit that removed 2 list items from 2 different expected ARIA snapshots.</p> <p>But tests continued to pass!?! &#129327;</p> <p>Found the root cause:</p> <blockquote><p> WTF, the snapshot tests still passed?!</p> <p>Investigated, and turns out this is because <a href="https://playwright.dev/docs/aria-snapshots#strict-matching">Playwright defaults to saying "yep, this is valid", if the expectation is a <em>subset of the actual</em>! &#129327;&#129760;</a></p> <p>Looks like we need <a href="https://playwright.dev/docs/aria-snapshots#strict-matching:~:text=deep%2Dequal%3A%20Matches%20if%20the%20children%20exactly%20match%20the%20specified%20list%20in%20order%2C%20including%20nested%20children"><code>/children: deep-equal</code></a> everywhere? &#128517;</p> <p>Out of scope here, opening new issue for it &hellip; </p></blockquote> <p>This is that issue.</p> <h3 id="proposed-resolution">Proposed resolution</h3> <p>Update all Playwright ARIA snapshot matching tests to use strict equality, or preferably, configure this systemwide so that we can't forget to specify <code>/children deep-equal</code>.</p> <p>(Basically: strictest by default, opt-in to loosening.)</p> <h3 id="ui-changes">User interface changes</h3> > Related issue: [Issue #3543770](https://www.drupal.org/node/3543770)
issue