Alternative approach to unique queue using upsert
>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3379798. --> Reported by: [ericgsmith](https://www.drupal.org/user/1649952) Related to !4 >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p>Hello - and thank you for this module! We have been using the <code>database_unique</code> and it greatly improved the default database queue from the purge module.</p> <p>We found that in certain situations checking each item individually in <code>findItem</code> was causing a bottleneck when multiple requests were received in a short time that invalidated the same cache tags (which in combination with purge_queuer_url) which cause thousands of requests to check duplicate items.</p> <p>My colleague suggested we try an alternative approach by creating a unique identifier on the queued item and using upsert queries to avoid the individual <code>findItem</code> calls.</p> <p>In our scenario, the ideal situation would be the ID of the URL from the url registry would flow through as the unique id of the invalidation (<a href="https://www.drupal.org/project/purge_queuer_url/issues/2941913#comment-15183007">https://www.drupal.org/project/purge_queuer_url/issues/2941913#comment-15183007</a>) but working with what the framework offers I have hashed the 2 values used by the <code>database_unique</code> queue to store as the unique index.</p> <h4 id="summary-steps-reproduce">Steps to reproduce</h4> <p>We have been testing this scenario with the followign setup:</p> <ol> <li><code>purge_queuer_url</code> enabled and add thousands of items into the registry with the <code>media_list</code> tag</li> <li>Installed and enabled <code>media_bulk_upload</code> - this module gives a quick way to produce multiple batch requests that invalidate <code>media_list</code></li> </ol> <p>I am adding a patch here in case you / anybody has a need for exploring an alternative approach to queues.</p>
issue