Draft: Issue #3082916 Consider all routing keys when creating and item in a queue

assigned to @zach.bimson

@cmlara its been such a long time since i did the original solution for this im now 100% on the motivation other than it felt like something that needed doing?

Honestly im not sure why you'd have more than one routing key on a queue where you WOULDN'T want to publish the message once for each key?

Im not sure i understand a usecase for having the routing_key be an array outside of this? Does this feel like something we even need?

Please do enlighten me if you can, im a bit lost on old me here

Note: I'm basing this off memory and my short notes on the issue, it is possible there are slight errors, however I believe overall the concept of below is accurate:

As it currently stands creating a queue called 'fanout_a_b' with routing_keys 'queue_a' and 'queue_b' will create create a queue that listens for 'fanout_a_b','queue_a' and 'queue_b'.

This current code is great if you want a queue that will listen to a message addressed to multiple locations to process (assuming RabbitMQ is utilizing the fanout exchange). Any external sender can send to 'fanout_a_b','queue_a' or 'queue_b' and Drupal will be able to claim a single copy of it with claimItem().

Currently when submitting a message with createItem() we will only submit it to 'fanout_a_b' (queues listening to 'queue_a' and 'queue_b' never get a copy). The feature request is that 'queue_a' and 'queue_b' also need a copy of the message.

The new code adds 2 new additional message sends to 'queue_a' and 'queue_b' for 3 messages injected into RabbitMQ for processing. Assuming the standard Fanout routing exchange on RabbitMQ the (Drupal created) 'fanout_a_b' queue receives 1 copy for each tag for a total of 3 copies of the same message and any other queue in RabbitMQ listening receives only one copy for each key they subscribe to.

Drupal may not even need to do anything with the message in which case the 'fanout_a_b' queue gets 3 copies that are never claimed and as such continue to backup inside the RabbitMQ server.

This is where I think we (likely) want some separation between the receive routing keys (those used with createQueue()) and the send key routing (those that we send message in createItem()).

I'm not sure if its more likely with multiple routing keys that a queue would be sending to more and listening to less, or vice versa, however since the current routing key handles the RX side, it seems (from a BC standpoint) that having a separate setting for the TX routing keys is easier to implement.

As for the root question, is this feature actually needed. I'm not 100% sure, I haven't needed it yet, however my messaging needs are certainly not complex (I have mostly used RabbitMQ to queue up remote long running jobs on remote hosts and file off a new message with the results for Drupal to claim later).

Could someone with much more complex needs, maybe?

At the moment this could be worked around by 3rd party code doing

$queue_a->createItem($data);
$queue_b->createItem($data);

However there isn't anything stopping an (accidental) claimItem() on those queues from inside Drupal since we make them available through the interface and because we register the queue in createQueue we have to have an exact match for the queue definitions just to send a message.

I haven't thought the following statement out fully, however it comes to mind that perhaps what we actually want is the ability for queues to be 'publish only' where we never register a subscription queue. We wouldn't need to add the 'send to each key' feature in that case (leave that to 3rd party code to inject a message into each queue) we would just need to prevent claiming items and queue creation on those.

(Hopefully this makes sense)

Draft: Issue #3082916 Consider all routing keys when creating and item in a queue

Merge request reports

Activity