>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #2568147. --> Reported by: [Mixologic](https://www.drupal.org/user/391689) >>> <p>WimLeers brought up a good point, that we may be disguising progress by not showing assertion counts.</p> <p>[04:46:27] <wimleers> Mixologic: oh btw<br> [04:46:39] <wimleers> Mixologic: I know the "assertions count" vs "test count" thing was discussed a long time ago somewhere<br> [04:46:47] <wimleers> Mixologic: But thinking about it more, that's actually a regression<br> [04:46:56] <mixologic> WimLeers: really?<br> [04:47:09] <wimleers> Mixologic: Because the same number of test failures can hide an increase or decrease in number of failed assertions<br> [04:47:20] <wimleers> Mixologic: So either not showing progress from patch N to N+1<br> [04:47:24] <wimleers> Mixologic: Or not showing regressions<br> [04:48:20] <wimleers> Mixologic: e.g. <a href="https://www.drupal.org/node/2476947#comment-10322637">https://www.drupal.org/node/2476947#comment-10322637</a> (#133) + #134 (the next)<br> [04:48:23] <druplicon> <a href="https://www.drupal.org/node/2476947">https://www.drupal.org/node/2476947</a> =&gt; Convert "title" page element into a block <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-7"><a href="https://www.drupal.org/project/drupal/issues/2476947" title="Status: Closed (fixed)">#2476947: Convert "title" page element into a block</a></span> =&gt; 168 comments, 33 IRC mentions<br> [04:48:31] <wimleers> Mixologic: 10 vs 7 fails in PIFR, 2 vs 2 fails in DrupalCI<br> [04:48:54] <wimleers> It was definite progress, but DrupalCI's reporting masked that<br> [04:51:32] <mixologic> WimLeers: Hmm. I see what you're saying there, however I was also seeing that the assertion count isnt actually a fixed number. It appears to have some random variability in it.<br> [04:51:52] <wimleers> Mixologic: Very rarely, TBH<br> [04:52:21] <wimleers> Mixologic: The variability is only to the extent that if one assertion fails very badly and triggers a fatal because of the assertion logic, that the rest of the test doesn't run.<br> [04:52:29] <mixologic> WimLeers: Im pretty sure its every time, but its probably isolated to a few particular tests that do something random... so the "grand total" is random, but not the error to errors.<br> [04:52:53] <wimleers> ohhhh right<br> [04:52:59] <wimleers> Mixologic: the total number of assertions seems random<br> [04:53:04] <wimleers> Mixologic: that's what you mean, right?<br> [04:53:08] <mixologic> Yes,<br> [04:53:09] <wimleers> Mixologic: yes, that's definitely true<br> [04:53:14] <wimleers> Mixologic: But *failed* assertions are not<br> [04:53:36] <wimleers> Mixologic: <strong>So, it'd be great if DrupalCI could say "13K tests, 5 failed, with 53 failed assertions"</strong><br> [04:53:45] <wimleers> Mixologic: i.e. it can continue to use "tests" as the main unit<br> [04:53:54] <wimleers> Mixologic: but if it could also tell us the number of failed assertions, that'd be super helpful<br> [04:54:08] <mixologic> We're still keeping counts of those assertions, and I've got that data in the xml, I just need to figure out a way to expose it to jenkins..</mixologic></wimleers></wimleers></wimleers></wimleers></wimleers></mixologic></wimleers></wimleers></wimleers></mixologic></wimleers></wimleers></mixologic></wimleers></wimleers></druplicon></wimleers></wimleers></wimleers></wimleers></mixologic></wimleers></wimleers></wimleers></p> <p>This will be postponed until: <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-1"><a href="https://www.drupal.org/project/drupalci_testbot/issues/2568137" title="Status: Active">#2568137: Expose Assertion Count to Jenkins</a></span> is done.</p> > Related issue: [Issue #2568137](https://www.drupal.org/node/2568137)