In recent weeks we have gotten various reports of things taking a long time on git.drupalcode.org. GitLab’s internal Sidekiq queues were being saturated creating queue delays of up to 15 minutes for over an hour. Some of this is expected, queues are meant to queue, but the delay manifests in noticeable ways in the UI, like: * Commits not appearing right away * Forks taking a long time to complete * CI jobs taking time to start * And more, GitLab uses queuing heavily With Claude-assisted log analysis, we have some reasonable-looking explanations, times are earlier today UTC: > Every push to `main` on project/drupal costs a \~97–100 minute `UpdateMergeRequestsWorker` run, all day, every day: 03:32, 03:56, 05:18, 05:31, 08:54... remarkably consistent at 5,800–6,100s each. The afternoon "incident" was simply what happens when commits land faster than refreshes complete. Subtracting durations from completion times: > > | push (start) | ref | duration | finished | > |--------------|-----|----------|----------| > | 14:32:30 | main | 10,869s | 17:33 | > | 14:39:08 | main | 5,873s | 16:17 | > | 14:42:56 | main | 10,263s | 17:33 | > | 14:52:23 | main | 5,598s | 16:25 | > | 15:03:49 | 11.x | 2,672s | 15:44 | > > catch landed four commits to main in a 20-minute window plus one to 11.x — a completely normal committer batch — which stacked **five concurrent refreshes**. The degradation onset at 14:43 is the third push arriving; the two earliest runs bloated from the \~6,000s baseline to \~10,900s under mutual contention. And the system recovered by \~16:25 while two refreshes were _still running_ until 17:33 — so one or two concurrent refreshes are absorbable background load; four or five are a site-wide event. The morning runs were serial by luck of commit spacing, which is why the day started healthy. That tracks with what I’ve seen in the Sidekiq dashboard > Practical mitigation you can ship this week: **serialize the refreshes.** The pushes have distinct args (oldrev/newrev), so deduplication never applies — but routing does: > > ``` > sidekiq['routing_rules'] = [ > ['worker_name=UpdateMergeRequestsWorker,MergeRequestResetApprovalsWorker', 'mr_refresh'], > ['*', 'default'] > ] > sidekiq['queue_groups'] = ['default,mailers', 'default,mailers', 'mr_refresh,default,mailers'] > ``` > > Ideally with low concurrency on the third process so refreshes run one or two at a time. That converts the pathological case (five refreshes fighting, everything degraded for 90 minutes) into the benign one you already tolerate daily (serial \~100-minute runs, system healthy throughout). The tradeoff — merge-detection and MR-widget freshness lagging further behind during commit batches — is real but marginal given the baseline is already an hour and a half. > > Beyond that, the levers are the ones we've identified: open-MR count against main/11.x is the linear factor in that 100 minutes (≈1.4s per MR), so stale-MR triage pays off directly