>>> [!note] Migrated issue <!-- Drupal.org comment --> <!-- Migrated from issue #3321961. --> Reported by: [hestenet](https://www.drupal.org/user/54034) >>> <h2>Transcript</h2> <h2>This meeting:&#10148; Is for core developers, initiative contributors, the Drupal Association and anyone interested in the initiative.&#10148; Usually happens every other Wednesday at 11:00 PT / Thursday 06:00 UTC&#10148; Is done over chat.&#10148; Happens in threads, which you can follow to be notified of new replies even if you don&rsquo;t comment in the thread. You may also join the meeting later and participate asynchronously!&#10148; Lots of threads get posted all at once! Don't worry - start at the top and then jump into the threads that most interest you.&#10148; Has a public agenda anyone can add to in the issue&#10148; *Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a &#128100; emoji. To take a comment or thread off the record, start with a &#128683; emoji.ping: @mixologic @FeyP @jurgenhaas (go to the issue of the next meeting to add or remove yourself from the list)0&#65039;&#8419; Who is here today? Comment in the thread below to introduce yourself!</h2> <table> <tr> <td>hestenet (he/him)</td> <td>Tim from the DA organizing our threads.</td> </tr> <tr> <td>markdorison</td> <td>Hi! Mark from Chromatic here. :blob_wave:</td> </tr> <tr> <td>FeyP</td> <td>Whoops, posted to a thread, but didn't say hello yet... Hi, I'm Patrick.</td> </tr> <tr> <td>quietone</td> <td>Hi, catching up</td> </tr> <tr> <td>saschaeggi</td> <td>Sascha from Gitlab :wave:</td> </tr> <tr> <td>hestenet (he/him)</td> <td>:clock11: Hi everyone, it's that time again - give me just a moment to get organized...</td> </tr> </table> <h2>1&#65039;&#8419; Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.</h2> <table> </table> <h2>2&#65039;&#8419; Postmortem of GitLab service degradation</h2> <table> <tr> <td>hestenet (he/him)</td> <td>The very short version of this is:As part of our effort to get ready for migrating issues, we wanted to implement a version of our collaborative by default workflow plan - starting by bulk updating all users to have developer access in the issue forks group. [#3250923]This seemed workable at first, until we ran into scaling issues when attempting the bulk update on production. The scaling issues are described here: <a href="https://gitlab.com/gitlab-org/gitlab/-/issues/381745In">https://gitlab.com/gitlab-org/gitlab/-/issues/381745In</a> essence, the number of enqueued operations Gitlab's sidkiq queues had to handle increased multiplicatively with the number of users being added, and got slower and slower. We caught it at something like 10% of the batch - but when we went to reverse it we discovered the reverse operation had the same scaling problem. With some clever rerouting of some traffic to the beefy AWS replica machine, and some very careful batching of requests and scripting of killing superfluous queue items, we managed to undo the bulk update and restore service. This took most of a week from initial migration attempt to final clean up. (edited)</td> </tr> <tr> <td>hestenet (he/him)</td> <td>As a result, we're obviously going to have to try something new for collaboration, which I'll talk about in the following thread.</td> </tr> <tr> <td>hestenet (he/him)</td> <td>EDIT: should have posted these in the following threadRelated issues in need of update as of the change in plans:<span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/infrastructure/-/work_items/3317273" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/infrastructure/-/work_items/3317273</a></span>&nbsp;(edited)</td> </tr> <tr> <td>larowlan</td> <td>#Hugops</td> </tr> <tr> <td>FeyP</td> <td>Thanks for all your work during the last weeks and for the updates along the way in the infrastructure channel and the detailed post-mortem here!&nbsp;(edited)</td> </tr> </table> <h2>3&#65039;&#8419; Consequences for our proposed collab-by-default workflow</h2> <table> <tr> <td>hestenet (he/him)</td> <td>This was discussed in pasting while we were trying to put out the GitLab fire, but because of the scaling issues above, we've realized we won't be able to simply grant all user developer access in a single group in order to collaborate.</td> </tr> <tr> <td>hestenet (he/him)</td> <td>The most likely solution/alternative to this is:Go ahead and use personal workspaces for forks instead. Use our own bot or a modified GitLab Triage-bot to add commands that grant access to collaborate - in a similar fashion to the buttons on d.o issues right now.</td> </tr> <tr> <td>hestenet (he/him)</td> <td>These are the issues where we'll need to document the updated plan (we still haven't totally validated it, either)</td> </tr> <tr> <td>hestenet (he/him)</td> <td><span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/infrastructure/-/work_items/3317273" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/infrastructure/-/work_items/3317273</a></span></td> </tr> <tr> <td>FeyP</td> <td>That's a tough decision to make.I have some sympathy for the "personal workspace" approach. That's the way it usually works in GitLab (and GitHub) and it will be easier for people new to the community to contribute when they are already familiar with the workflow. Also, using a more out of the box workflow should be more sustainable with regards to leveraging issue fixes/new features from upstream, support in case of potential problems and probably also less work for our infrastructure team.Then again, it might get messy when each contributor creates their own issue forks and MRs for the same issue. Might be easy to miss some changes in the process for more complex issues.With a bot, we could probably maintain an approach that is more consistent with our original goals. The downside is that it is again kind of a "special" workflow that new contributors would have to discover/learn. Also, maintaining the bot across major updates might require some extra work.</td> </tr> </table> <h2>4&#65039;&#8419; GitLabCI</h2> <table> </table> <h2>4&#65039;&#8419; 1&#65039;&#8419;&nbsp; GitLab CI templates are available directly on git.drupalcode.org - has anyone had much chance to try them?</h2> <table> <tr> <td>hestenet (he/him)</td> <td>Reminder of the instructions:When adding or editing a gitlab-ci.yml file with the web IDE, you have the option to choose from the custom templates. The current one I recommend starting with is actually called template.gitlab-ciThere is a lot more to do - these templates should basically work, but they will still undergo a lot of changes as we try and do things like improve concurrency and tune up the jobs. Your feedback on improving the templates will be appreciated.Merge requests accepted here:<a href="https://git.drupalcode.org/project/gitlab_templatesOr">https://git.drupalcode.org/project/gitlab_templatesOr</a> (for more experimental stuff) here: <a href="https://gitlab.com/drupal-infrastructure/gitlab-acceleration/gitlabci/file-templatesThe">https://gitlab.com/drupal-infrastructure/gitlab-acceleration/gitlabci/file-templatesThe</a> project page/issue queue: drupal.org/project/gitlab_templates</td> </tr> <tr> <td>hestenet (he/him)</td> <td>@phenaproxima I know you were going to try this out for Lightning, but it took us a bit to get you opted in while we were distracted putting out the GitLab fire. Any chance to look yet?</td> </tr> <tr> <td>hestenet (he/him)</td> <td>I hope the inline documentation on the template file is reasonably clear, but that would be a good area for feedback.</td> </tr> <tr> <td>moshe</td> <td>I got it to go green ... I have a bunch of notes I need to turn into issues</td> </tr> <tr> <td>hestenet (he/him)</td> <td>Progress! It was absolutely expected that this would all still need work, so looking forward to issues so we can knock things out :flex:</td> </tr> </table> <h2>4&#65039;&#8419; 2&#65039;&#8419; Known issue: GitLabCI templates need nightwatch testing configuration</h2> <table> <tr> <td>hestenet (he/him)</td> <td>This is on my list to try to look at.&nbsp;(edited)</td> </tr> </table> <h2>4&#65039;&#8419; 3&#65039;&#8419; Known issue: GitLabCI templates need parallelization</h2> <table> <tr> <td>hestenet (he/him)</td> <td>Looks like @chrisfromredfin and team are going to look into this, as they play with using GitlabCI for Project Browser</td> </tr> </table> <h2>5&#65039;&#8419; General next steps for performance and scaling of GitLab infra</h2> <table> <tr> <td>hestenet (he/him)</td> <td>Our current installation uses the GitLab Omnibus installation type, with as minimal changes as we can make so that we're on a relatively standard set up.The primary and SSH traffic handler is still the instance that lives in our virtualized environment at the Oregon State Open Source Lab.The secondary and HTTPS traffic handler is a very large AWS instance, which we've also tuned up for more simultaneous queue processing in the course of resolving the issues last week.Our likely next steps will be:Make the larger AWS instance the primary, and the OSL instance the replicaLook at deploying another replica on AWS in EuropeHowever - as of right now - performance seems really strong.If you have seen specific performance issues in the last couple of days, we want to know about them. (cc @wimleers (he/him))</td> </tr> </table> <h2>6&#65039;&#8419; Contribution credit:@fjgarlin is building this.Including process for moving contribution record to a dedicated content type, so it can take all prior d.o issue credit, and also be linked by via webhook to the new Gitlab issues (which, reminder, he has already built migration code for)</h2> <table> <tr> <td>hestenet (he/him)</td> <td>We'll add a child issue to <span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/infrastructure/-/work_items/3227737" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/infrastructure/-/work_items/3227737</a></span> with the details</td> </tr> <tr> <td>fjgarlin</td> <td>Added <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-1"><a href="https://www.drupal.org/project/infrastructure/issues/3322116" title="Status: Active">#3322116: Contribution records revamp</a></span>&nbsp;(edited)</td> </tr> <tr> <td>hestenet (he/him)</td> <td>:thankful:</td> </tr> </table> <h2>I&rsquo;m refocusing a bit on simplifying Drupal.org, <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-7"><a href="https://www.drupal.org/project/project_issue_file_test/issues/3322002" title="Status: Closed (fixed)">#3322002: Remove per-issue feature branch testing</a></span> is an upcoming functionality removal. I suspect not many maintainers are using it.</h2> <table> </table> <p>Participants:</p> <p>hestenet, markdorison, FeyP, quietone, saschaeggi, larowlan, moshe, fjgarlin</p>