>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3545918. --> Reported by: [robin.houtevelts](https://www.drupal.org/user/3500121) Related to !3 >>> <h3 id="summary-problem-motivation">Problem/Motivation</h3> <p>The <code>drush queue:run-all</code> command is an excellent tool for processing all queues via a background daemon. Setting this up with a process manager like <code>systemd</code> provides a robust way to handle background tasks continuously.</p> <p>However, the current implementation processes <em>all</em> queues with a defined worker. In complex applications, it's often desirable to have more granular control over queue processing. For instance:</p> <ul> <li>A high-priority queue (e.g., sending transactional emails) might need one or more dedicated workers to ensure items are processed instantly, without waiting for a long-running, low-priority task (e.g., generating large reports) to finish.</li> <li>A high-volume queue could "starve" other queues, preventing them from being processed in a timely manner if they all share the same worker pool.</li> <li>Different queues might have different resource requirements, and dedicating workers allows for better resource management on multiple servers/containers.</li> </ul> <p>This functionality is common in other queueing systems. For example, Symfony's Messenger component allows consumers to be limited to specific queues: <code>bin/console messenger:consume async --queues=fasttrack1 --queues=fasttrack2</code>.</p> <p>This proposal suggests adding a similar capability to <code>drush queue:run-all</code> to allow for the creation of dedicated daemons per queue, or a group of queues.</p> <h4 id="summary-steps-reproduce">Steps to reproduce</h4> <p>This is a feature request, so there are no steps to reproduce a bug.</p> <h3 id="summary-proposed-resolution">Proposed resolution</h3> <p>I propose adding two new, mutually exclusive options to the <code>drush queue:run-all</code> command:</p> <ol> <li> <strong><code>--queues</code></strong>: A comma-separated list of queue machine names to process. When this option is provided, the command will <em>only</em> claim items from the specified queues.<br> <br><br> <strong>Example:</strong> Create a dedicated worker for geocoding and sitemap submission. <pre>drush queue:run-all --daemon --queues=geocoder,simple_sitemap_engines</pre></li> <li> <strong><code>--exclude-queues</code></strong>: A comma-separated list of queue machine names to exclude from processing. When this option is provided, the command will process all available queues <em>except</em> for the ones specified. This is useful for creating a "main" or "catch-all" daemon that ignores the queues handled by dedicated workers.<br> <br><br> <strong>Example:</strong> Create a general worker that ignores the queues handled by dedicated daemons. <pre>drush queue:run-all --daemon --exclude-queues=geocoder,simple_sitemap_engines</pre></li> </ol> <p>These options would allow for a flexible and powerful daemon configuration using <code>systemd</code>. For example, one could define multiple services:</p> <ul> <li><code>queue-runner-high-priority.service</code>:<br> <code>ExecStart=/path/to/drush queue:run-all --daemon --memory-limit=60% --queues=high_priority_queue,another_fast_queue</code> </li> <li><code>queue-runner-main.service</code>:<br> <code>ExecStart=/path/to/drush queue:run-all --daemon --memory-limit=60% --exclude-queues=high_priority_queue,another_fast_queue</code> </li> </ul> <p>This setup ensures high-priority tasks are handled immediately by their own worker, while all other tasks are handled by a general worker, improving the overall responsiveness and reliability of the system.</p> <h3 id="summary-remaining-tasks">Remaining tasks</h3> <ul> <li>Implement the <code>--queues</code> and <code>--exclude-queues</code> options in the <code>QueueRunAllCommand</code>.</li> <li>Add validation to ensure the options are mutually exclusive.</li> <li>Update the command's help text to document the new options.</li> <li>Add unit/integration tests to cover the new functionality.</li> <li>Code review and merge.</li> </ul> <h3 id="summary-ui-changes">User interface changes</h3> <p>None. This is a change to a Drush command-line tool.</p> <h3 id="summary-api-changes">API changes</h3> <p>None. This change introduces new, optional options to a Drush command. It is a backwards-compatible addition.</p> <h3 id="summary-data-model-changes">Data model changes</h3> <p>None. This does not alter the database schema or configuration data model.</p>