Duplicate queued jobs keep getting added

Migrated issue

Reported by: jhodgdon

This is a follow-up to
#774682: Convert to use Drupal Queue

Since converting to Drupal Queue, the number of jobs in my test site's queue keeps growing. (I only realized this due to implementing a fix for #1432050: Show parsing status on branch admin page, and seeing the number growing instead of shrinking each cron run.)

The reason is that when we were using Job Queue, we were calling job_queue_add() with a $no_duplicates = TRUE argument, which made sure that no duplicate jobs were added. But now that we are using Drupal Queue, duplicate jobs are added each cron run, because each cron run the parser checks to see what needs updating (what files have been touched since the last time they were parsed), and adds all of them to the Drupal Queue.

There doesn't seem to be a way to tell Drupal Queue to avoid adding duplicates, unfortunately. The Drupal Queue that we are using is a backport of the Drupal 7 core queue, and it doesn't have this ability.

We might need to go back to Job Queue... The only other option I can think of is keeping track in the API module. This is actually feasible, because we could put queued items in a table and then when the parse function is called, remove them again. But it seems a little pointless then to use Drupal Queue if we have to keep track ourselves?

Anyway, the current state of things is *not* OK. It would never finish parsing.