>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3466306. --> Reported by: [luke.leber](https://www.drupal.org/user/3509746) Related to !41 !40 >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <div class="messages warning"> This is a public follow-up for <a href="https://security.drupal.org/node/180370">https://security.drupal.org/node/180370</a> following the 2 week waiting period for both maintainers and security team members to change the issue status away from "Needs public followup". </div> <p>Acquia Search has a potential to become a contributor to temporary denial of service conditions on cache clear due to excessive locking, which can lead to exhaustion of PHP processes. On cold caches, multiple blocking HTTP requests are made from the Drupal application to Acquia hosted services, each of which have nondeterministic timing in their responses. Acquia Search in particular has the potential to lock a PHP process for up to 90 seconds.</p> <p>In <code>\Drupal\acquia_search\AcquiaSearchApiClient::getSearchIndexes</code> we see the following code:</p> <pre>&nbsp;&nbsp;&nbsp; $lock = 'acquia_search_get_search_indexes';<br>&nbsp;&nbsp;&nbsp; if (!$this-&gt;lock-&gt;acquire($lock)) {<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $this-&gt;lock-&gt;wait($lock);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // Throw an exception after X amount of seconds.<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (!$this-&gt;lock-&gt;acquire($lock)) {<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; throw new \Exception("Couldn't acquire lock for $lock in less than 30 seconds.");<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }<br>&nbsp;&nbsp;&nbsp; }</pre><p>Picking this apart, the first call to <code>$this-&gt;lock-&gt;acquire($lock))</code> has the potential to block the process for up to 30 seconds. Next, <code>$this-&gt;lock-&gt;wait($lock)</code> has the potential to block the process for an <em>additional 30 seconds</em>. Finally, the second call to <code>$this-&gt;lock-&gt;acquire($lock)</code> can block for <em><strong>another 30 seconds</strong></em>. In total, it seems that this code has the latent potential to halt a PHP process for up to 90 seconds.</p> <p>We have seen this exception thrown recently on deployments right after <code>drush cr</code> steps.</p> <h4 id="summary-steps-reproduce">Steps to reproduce</h4> <p>With a large site under heavy load, flush caches. Sometimes, witness that PHP processes are exhausted and that <code>Couldn't acquire lock for acquia_search_get_search_indexes in less than 30 seconds.</code> exceptions appear in the error logs.</p> <h3 id="summary-proposed-resolution">Proposed resolution</h3> <p>Greatly reduce lock timing from the default 30 seconds to something more sane. What exactly is the API performance timing of the Acquia services that the HTTP calls are made to? Responses in 1 second? 3 seconds? Surely 30 second responses aren't to be expected from the Acquia services.</p> <p>- or -</p> <p>Only lock on one process and on subsequent requests, return immediately, perhaps making a call to <code>\Drupal::messenger()</code> with a user-friendly error message. I.e. "Acquia Search is not available right now, please try again in a few seconds."</p> <h3 id="summary-remaining-tasks">Remaining tasks</h3> <p>Discuss, open MR, commit, profit.</p> <h3 id="summary-ui-changes">User interface changes</h3> <p>Perhaps a new end-user facing error message.</p> <h3 id="summary-api-changes">API changes</h3> <p>TBD</p> <h3 id="summary-data-model-changes">Data model changes</h3> <p>None</p>