>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3105968. --> Reported by: [Mixologic](https://www.drupal.org/user/391689) >>> <p>In wrestling with the packinging pipeline architecture, It's become clear that a queue solves the connectivity issue between drupal.org and jenkins, and allows for events to happen when jenkins is offline, however we really need something closer to what drupalci provides us, which is a job database that we can query for status, and manipulate in a more fine grained way than we can with the supervisord queues. But we actually need both.</p> <p>When one of the two events that triggers packaging occurs (1. git push to a 'development' branch', or 2. A release is created on drupal.org), a new "queued" drupalorg_packaging_job will be created, which contains all the information that the packaging pipeline will need to process the job.</p> <p>Then, a message will be added to the supervisord queue, with only the drupalorg_packaging_job:job_id in it. The queuerunner's job is to attempt to contact jenkins, and kick off the jenkins pipeline, passing it the data from the drupalorg_packaging_job. (And setting the status of the job to "sent").</p> <p>The pipeline job on jenkins will then process all of the packaging steps. The first step sets the job to "running", and if all is successful, will set the status of the job to "complete" or "error" if it had any errors or issues along the way and was able to recognize that. </p> <p>In the event that the jenkins job dies and does not update the drupalorg_packaging_job database, we'll end up with jobs in a 'running' status.<br> After a configurable amount of time, we should assume that a "running" job has actually errored out, and either requeue it or trigger an alert. We might also ignore these and set them to some other status like "failed" in the event that the job is a branch and a newer commit has already been processed.</p> <p>The logic for creating a new drupalorg_packaging_job should also do some intelligent pre-aggregation. Essentially "do not create a job for a release_nid if a job already exists and is either 'queued' or 'sent'. This allow for multiple rapid pushes to a dev branch to not create multitudes of packaging jobs, but also, if one is already running, it would still queue another to run when it has completed.</p> <p>The job status' mirror PIFT identically, and semantically mean the same thing.</p> <p>The architectural difference is that with pift, we currently have cron_ci_send_<a href="http://www.drupal.org">www.drupal.org</a>, which polls the drupalorg database once a minute and sends jobs to dispatcher, and in this proposed architecture we'd have a supervisord queue that gets autoprocessed when something is ready to be sent - so a push vs a poll model. Polling jobs often create a lot of noise and can be difficult to track down where in their logs something occurred. A push job will yield something that only does work when there's work to be done, and since its only job is messaging, can handle disconnects and failures without any additional logic.</p> > Related issue: [Issue #3101993](https://www.drupal.org/node/3101993)