GitLab Acceleration Initiative meeting on 17 May, 2023
>>> [!note] Migrated issue
<!-- Drupal.org comment -->
<!-- Migrated from issue #3361089. -->
Reported by: [hestenet](https://www.drupal.org/user/54034)
>>>
<h2>Transcript</h2>
<h2>0️⃣ Who is here today? Comment in the thread below to introduce yourself!</h2>
<table>
<tr>
<td>hestenet (he/him)</td>
<td>Tim from the DA :wave::skin-tone-3:</td>
</tr>
<tr>
<td>markdorison</td>
<td>Mark from Chromatic here :blob_wave:</td>
</tr>
<tr>
<td>irinaz</td>
<td>hi, Irina Zaks, working from Bay Area</td>
</tr>
<tr>
<td>dww</td>
<td>Derek, freelance (potentially available) :wave:</td>
</tr>
<tr>
<td>tyler36</td>
<td>Hi</td>
</tr>
</table>
<h2>1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.</h2>
<table>
<tr>
<td>cmlara</td>
<td>Update on PATS investigation</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>(Sorry, can you remind me what that is? I'll get the thread together..)</td>
</tr>
<tr>
<td>cmlara</td>
<td>(next item)Will testing be enabled for issue forks or just main projects?Relating to <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-5"><a href="https://www.drupal.org/project/site_moderators/issues/3309915" title="Status: Closed (won't fix)">#3309915: Add testing configuration as an official task of a site moderator</a></span> and 'trouble' we have had in the past with untested projects.</td>
</tr>
<tr>
<td>Juraj Nemec</td>
<td>Is there an issue somewhere about the progress with D7 contrib testing on Gitlab CI? Or maybe just the brief summary what is the status / problems, ...</td>
</tr>
</table>
<h2>2️⃣ We're getting closer to being able to share a preview of GitLab issue management - @fjgarlin - will hopefully have a bit of a video demo in time for DrupalCon, and we may share something here sooner, we'll see.</h2>
<table>
</table>
<h2>2️⃣ Speaking of DrupalCon - if we're able to finish the global availability of GitLabCI - would there be folks willing to hang out in Contrib area and help people get set up on GitLab CI?</h2>
<table>
<tr>
<td>crasx</td>
<td>Could be a good bof if those are still open? I haven’t looked at the gitlab stuff recently but I’m happy to be a set of hands. I’ll be in the mentoring space, hopefully contrib space and mentoring space are close by</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>They are all on the 4th floor of the convention center in connected spaces, so should be pretty straightforward, I think</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>I'll probably spend some time on a 'how-to' during the drupal.org update panel</td>
</tr>
<tr>
<td>crasx</td>
<td>awesome. Excited to catch up on all your progress. This is such a great initiative</td>
</tr>
<tr>
<td>irinaz</td>
<td>@hestenet (he/him), is there a day/time set for such BoF or session</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>No BoF scheduled for this specifically - was just planning to claim a table in contributino</td>
</tr>
<tr>
<td>irinaz</td>
<td>I can be there to help folks navigate templates and setup, for me it is easier to have some time allocated</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>That'd be great</td>
</tr>
</table>
<h2>3️⃣ Threads for GitLab CI general availability</h2>
<table>
</table>
<h2>3️⃣ 1️⃣ Upgrading the GitLab runner configuration. The current configuration has been working for our initial testers, but really needs to be leveled up. Getting some help from @nnewton and @Max Whitehead hopefully, although they are also very busy helping with Rugged signing. (edited) </h2>
<table>
<tr>
<td>moshe</td>
<td>Would this address the multi-minute delay we sometimes see likeWaiting for pod gitlabci-prod/runner-jtjnatxc-project-22902-concurrent-0cjzwr to be running, status is Pending<br>
Unschedulable: "0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector."<br>
Waiting for pod gitlabci-prod/runner-jtjnatxc-project-22902-concurrent-0cjzwr to be running, status is Pending<br>
Unschedulable: "0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector."<br>
Waiting for pod gitlabci-prod/runner-jtjnatxc-project-22902-concurrent-0cjzwr to be running, status is Pending<br>
Unschedulable: "0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector."<br>
Waiting for pod gitlabci-prod/runner-jtjnatxc-project-22902-concurrent-0cjzwr to be running, status is Pending<br>
Unschedulable: "0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate."<br>
Waiting for pod gitlabci-prod/runner-jtjnatxc-project-22902-concurrent-0cjzwr to be running, status is Pending<br>
ContainersNotReady: "containers with unready status: [build helper]"<br>
ContainersNotReady: "containers with unready status: [build helper]"<br>
Running on runner-jtjnatxc-project-22902-concurrent-0cjzwr via gitlab-prod-runner-gitlab-runner-66679bc9bd-dbkx6...</td>
</tr>
<tr>
<td>moshe</td>
<td>Thats from <a href="https://git.drupalcode.org/project/keycdn/-/jobs/46961">https://git.drupalcode.org/project/keycdn/-/jobs/46961</a></td>
</tr>
<tr>
<td>nnewton</td>
<td>No, delays like that are spot requests taking a bit to spin up vms</td>
</tr>
<tr>
<td>nnewton</td>
<td>i.e. what is happening in the background is its requesting a spot VM and aws is fulfilling that and the speed of that is defined by spot availability. To help this we could keep spot VMs spun up, but right now there is not enough activity to validate cost for that</td>
</tr>
<tr>
<td>moshe</td>
<td>Got it. Thanks for explaining. Once demand warrants, is it a big dev effort to switch to the "keep VMs spun up" mode?</td>
</tr>
<tr>
<td>nnewton</td>
<td>No, just config for the autoscaler to keep a min number up</td>
</tr>
<tr>
<td>nnewton</td>
<td>Right now it scales down to 0 and is mostly at 0 throughout the day</td>
</tr>
</table>
<h2>3️⃣ 2️⃣ Solving the 'run pipelines for MRs opened by non-members of projects'akaWill testing pipelines be enabled for the forks themselves, or just the parent projects when MRs are opened against them. (edited) </h2>
<table>
<tr>
<td>hestenet (he/him)</td>
<td>Want to designate a test project where we can try out non-member MRs.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>@moshe should we use keycdn for this?</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>@cmlara - I think this thread is actually relevant to your second topic question</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>It's looking like due to how GitLab handles non-member permissions when triggering tests on upstream projects we may have to enable testing on fork projects, and not just the parent projects.<a href="https://git.drupalcode.org/help/ci/pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project ">https://git.drupalcode.org/help/ci/pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project </a>(edited)</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Unless there is a way we haven't found properly just yet to trigger upstream MR tests if the MR is opened by a non-member.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>(Maybe something more we can do with 'downstream' or 'upstream' keywords in workflow config)</td>
</tr>
<tr>
<td>moshe</td>
<td>@hestenet (he/him) I dont see a way to avoid pipelines running in forks, unless we want to go all the way back to the proposal to use personal forks instead of shared forks. Combiend with a bot that will grant access to any personal fork, I think this is a solid approach. CI minutes will be tracked by individual, which is the right way IMO. And the WebIDE automatically works in the personal fork, etc. Anyway, feel free to shut down this recurring conversation as unhelpful.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>No worries - there's a bot user planned for the access grants either way - whether the forks are personal or in a shared namespace.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>But can you clarify how personal forks avoid having pipelines running in forks rather than parents?</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Or are you just saying it would still run in forks, they'd just be easier to track against users?</td>
</tr>
<tr>
<td>cmlara</td>
<td>Sounds like a 'happy accident' to me as it relates to the issue I linked, Makes it irrelevant if a maintainer has not yet enabled testing as tests can be proven in the fork before merging.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>That issue is: <span class="drupalorg-gitlab-issue-link project-issue-status-info project-issue-status-5"><a href="https://www.drupal.org/project/site_moderators/issues/3309915" title="Status: Closed (won't fix)">#3309915: Add testing configuration as an official task of a site moderator</a></span></td>
</tr>
<tr>
<td>moshe</td>
<td>But can you clarify how personal forks avoid having pipelines running in forks rather than parents?They are still in forks, its just that the minutes are charged to a user, not to a big group. I'm not too clear on what the downsides are with pipelines in forks. Might be good to list them. (edited)</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>The downside is we were trying to manage CI limits per-project rather than per group or per user - it looks like we'll just have to rethink our strategy on that.</td>
</tr>
<tr>
<td>moshe</td>
<td>Gotcha. What do you think about turning on pipelines in forks so we get some experience with them?</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>That's how I'm leaning.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>The issue is we don't have our bulk permission update stuff written yet for the projects enabled though</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>So how we do that as brand new forks get created is going to be a bit of a trick</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Could do it manually for one or two in order to test maybe.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>But I'd also like to do some more reading.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>(Sorry @drumm, I may have cut you off)</td>
</tr>
<tr>
<td>drumm</td>
<td>This needs an issue summary somewhere, following all the threads is impractical</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Good point - I can start one up.</td>
</tr>
<tr>
<td>moshe</td>
<td>We must already have a way to create new forks and configure them as needed. I would think we add on to that so the checkbox for public pipelines is enabled. But in any case, I would agree that enabling a couple by hand makes sense.</td>
</tr>
<tr>
<td>drumm</td>
<td>No, we don’t want to be doing that sort of testing in production and setting off integrity checks we have making sure projects are configured as expected.</td>
</tr>
<tr>
<td>drumm</td>
<td>I still haven’t been able to understand what specifically is not working.</td>
</tr>
<tr>
<td>moshe</td>
<td>Great that we have integrity checks! I can summarize if @hestenet (he/him) gets bogged down with that issue summary.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Issue summary as best I understand it. Please update with additional explanation where helpful:<span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/drupalorg/-/work_items/3361106" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/drupalorg/-/work_items/3361106</a></span></td>
</tr>
<tr>
<td>drumm</td>
<td>Thanks, that’s much better. With re-reading <a href="https://git.drupalcode.org/help/ci/pipelines/merge_request_pipelines.md">https://git.drupalcode.org/help/ci/pipelines/merge_request_pipelines.md</a> and looking over a second project <a href="https://git.drupalcode.org/project/scheduler/-/merge_requests">https://git.drupalcode.org/project/scheduler/-/merge_requests</a>, I think I get it now.</td>
</tr>
<tr>
<td>drumm</td>
<td>The downside is we were trying to manage CI limits per-project rather than per group or per user - it looks like we’ll just have to rethink our strategy on that.It looks like we’re almost certainly on our own for CI usage monitoring. Since we have to build that custom, we can sum up the minutes however we want, given enough effort. Fork CI usage could be rolled up into the project usage.</td>
</tr>
<tr>
<td>dww</td>
<td>Thanks for the useful thread, folks. :pray: Following 3361106 now.</td>
</tr>
<tr>
<td>moshe</td>
<td>Perhaps conf call about this would be helpful? I'm concerned this will delay the GA of Gitlab CI so want to keep momentum. I know folks are prepping for Drupalcon so if we have to delay until then thats understandable.</td>
</tr>
<tr>
<td>drumm</td>
<td>The only real hard blocker to GA is making sure Kubernetes/etc are set up to be a bit more reliable. The resourcing it has now is very much a proof of concept that happens to work a lot of the time. Otherwise, since adoption requires work from maintainers to set up, a gradual adoption to keep finding the gaps is ideal. In the next few weeks, we should identify what’s a blocker to turning off DrupalCI and motivating adoption.</td>
</tr>
<tr>
<td>moshe</td>
<td>You think we can go GA with non maintainer mrs unable to run pipeline?</td>
</tr>
<tr>
<td>drumm</td>
<td>I wouldn’t start telling everyone to move right away without a few issues fixed, including that one. But I wouldn’t let that stop us from letting maintainers give it a try without the opt-in process</td>
</tr>
<tr>
<td>drumm</td>
<td>The thing that would be best to catch early is any type of issue that would require many maintainers to go back and change their gitlab-ci.yml if there’s a general problem that wasn’t spotted. I don’t want to waste their time if we can avoid it. Things we can iteratively improve ongoing without requiring maintainer work can keep happening when ready</td>
</tr>
<tr>
<td>moshe</td>
<td>We have accomplished that. The whole template is internal to us now.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Just repeating what Drumm said, I guess, but to summarize:We still think we can provide GA to GitLabCI before DrupalConWe should do an other audit of potential blockers::checkbox: We have finished making the template completely updatable internally through the include file structure - that part should be fine. :checkbox-no: Blocker: We are finishing updating the resource request config for the runners:checkbox-no: Blocker: Still need to write the bulk update script for updating all projects to enable access. :checkbox-maybe: Non-blocker: We should do a quick update of the current state of the variables, :checkbox-maybe: Non-blocker: but important follow-up - usage monitoring of AWS resources:checkbox-maybe: Non-blocker: but important followup: Make final decision on just enabling CI in forks or using a bot as the alternate solution for triggering tests from MRs(edited)</td>
</tr>
<tr>
<td>drumm</td>
<td>And “GA” should be considered a soft launch until known dead ends are resolved. Like <span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/drupalorg/-/work_items/3361106" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/drupalorg/-/work_items/3361106</a></span></td>
</tr>
<tr>
<td>moshe</td>
<td>For bullet 3, Would you consider doing a few projects by hand so we can see where the next hurdle is (if any).</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Aren't the current manual opt ins exactly that, or are you thinking something else?</td>
</tr>
<tr>
<td>drumm</td>
<td>Bulk updates are not a huge deal, they just take time since its 1-2 http requests per project. They are somewhat routine as GitLab changes</td>
</tr>
<tr>
<td>moshe</td>
<td>I mean that we fix a few issue forks so their contributors (i.e. non-maintainers) can get pipelines run</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Oh bullet 6 really</td>
</tr>
<tr>
<td>drumm</td>
<td>It looks like none of the work mentioned in either issue has been started, so no, we’re not going to do testing in production</td>
</tr>
<tr>
<td>moshe</td>
<td>I'm actively trying to make progress on <span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/gitlab_templates/-/issues/3361330" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/gitlab_templates/-/issues/3361330</a></span> a prerequisite? That seems like an unrelated issue and a lower priority issue since it only affects maintainers. Nobody else can bypass MRs and commit right to the contrib project. (edited)</td>
</tr>
<tr>
<td>drumm</td>
<td>They aren’t blockers, but certainly adjacent. If we decide to test all branches instead of those ending in .x , that would affect <span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/drupalorg/-/work_items/3361106" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/drupalorg/-/work_items/3361106</a></span>. And GitLab 16 added a button for syncing a fork with its parent project. Would that kick off redundant tests in the fork?</td>
</tr>
<tr>
<td>drumm</td>
<td>Knowing what branches are tested removes an unknown thing that could affect what we choose</td>
</tr>
<tr>
<td>cmlara</td>
<td>A key aspect to rember:Committers can always override everything in the templates, all your doing is setting defaults as part of the template process. So don't think of 3361330 as the final answer to anything.</td>
</tr>
<tr>
<td>moshe</td>
<td>For sure.I tried to summarize <span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/gitlab_templates/-/issues/3361330" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/gitlab_templates/-/issues/3361330</a></span>#comment-15075939. I probably wont do the work there, because I suck at regex and its not an especially urgent problem in my mind. The maintainer can always open an MR just like his contributors do.</td>
</tr>
<tr>
<td>moshe</td>
<td>They aren’t blockers, but certainly adjacent.OK, so I circling back. Can we enable pipelines on a few issue forks to confirm how much needs changing to fix point 6? Or we get started on a bot? This conversation is is easiest IMO with a conf call but Slack is OK too.</td>
</tr>
<tr>
<td>drumm</td>
<td>We have a dev server for experimental testing</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>We've got testing steps for this outlined in the issue, so myself or Fran or Neil or whoever has the next best opportunity will run through those to validate in the dev env.</td>
</tr>
<tr>
<td>drumm</td>
<td>I’d really like to test this with <a href="https://about.gitlab.com/releases/2023/05/22/gitlab-16-0-released/#update-your-fork-from-the-gitlab-ui">https://about.gitlab.com/releases/2023/05/22/gitlab-16-0-released/#update-your-fork-from-the-gitlab-ui</a> in dev, which we can update to GitLab 16. However, staging is offline pending an infrastructure migration, and I’d like a test environment on the same version as production since we’ll likely have one more upgrade on 15.*</td>
</tr>
<tr>
<td>drumm</td>
<td>@moshe in <span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/gitlab_templates/-/issues/3361330" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/gitlab_templates/-/issues/3361330</a></span>#comment-15075939 was #1 a typo? If all commits run pipelines, that’s redundant with #3</td>
</tr>
<tr>
<td>moshe</td>
<td>Woops, bad typo. Fixed.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>This pipeline just ran with the newly configured resource requests configuration for the cluster - which should be beefy as hell, and ensure that jobs aren't competing as much.This checks off item 2 on our general availability checklist.</td>
</tr>
</table>
<h2>3️⃣ 3️⃣ Performing the actual bulk update itself (edited) </h2>
<table>
</table>
<h2>3️⃣ 4️⃣ Non-blocker- but improved usage monitoring. (edited) </h2>
<table>
</table>
<h2>4️⃣ Update on Personal Access Tokens investigation per @cmlara (edited) </h2>
<table>
<tr>
<td>cmlara</td>
<td>This is [#3199831]Last discussion here on Slack was that needed to validate there wasn't anything in the API that users shouldn't have access to so we could enable them.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Oh - personal access tokens, gotcha</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>This is still a todo - I'm sorry about that. Any GitLab time I've had I've been putting on the CI side, so this hasn't hit the top of my list.</td>
</tr>
<tr>
<td>cmlara</td>
<td>Fair (and I get it, there are absolutely things that need to be done first.)I'll note the more we move onto GitLab, the more useful it is to have this sorted. I'm concerned were going to hit a tipping point where this starts making it harder on maintainers and contributors to work with their projects.I do think we should have this sorted before we consider allowing Issues to be handled on Gitlab (I've seen some talk about allowing some projects to opt in early) as we need it to replace the project_issues D.O. API (unless we plan on proxying that through D.O. in some manner)</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Noted - thanks for continuing to keep on top of this and provide reminders - it does help.</td>
</tr>
</table>
<h2>5️⃣ core gitlabci @xurizaemon @RoSk0 and I are hoping to spend some time at DrupalSouth sprint on Friday looking at the performance issue</h2>
<table>
<tr>
<td>moshe</td>
<td>Might be work giving personal access tokens to these folks so they can get a terminal.</td>
</tr>
<tr>
<td>andypost</td>
<td>making image size smaller should speed-up all jobs</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>@larowlan In addition to looking into runner terminal access - any other support y'all need before Friday, best case?</td>
</tr>
<tr>
<td>larowlan</td>
<td>I think that's it</td>
</tr>
<tr>
<td>dww</td>
<td>making image size smaller should speed-up all jobsIs there an issue for this?</td>
</tr>
<tr>
<td>andypost</td>
<td>@dww I bet there's no but x2-x3 possible</td>
</tr>
<tr>
<td>dww</td>
<td>Sounds great!</td>
</tr>
<tr>
<td>moshe</td>
<td>Not sure how caching works but this same image will be used on virtually all of our runners. Would not be wise to download it from docker hub every time.</td>
</tr>
<tr>
<td>andypost</td>
<td>@moshe docker hub has rate-limits, that's why Gitlab using own registry and proxy<a href="https://about.gitlab.com/blog/2020/10/30/mitigating-the-impact-of-docker-hub-pull-requests-limits/ ">https://about.gitlab.com/blog/2020/10/30/mitigating-the-impact-of-docker-hub-pull-requests-limits/ </a>(edited)</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>3️⃣ 5️⃣ (reserved)</td>
</tr>
</table>
<h2>6️⃣ Issue for working on D7 testing compatibility:[#3343810]per: @Juraj Nemec (edited) </h2>
<table>
<tr>
<td>hestenet (he/him)</td>
<td>Looks like this issue didn't exist already, so I've created one.</td>
</tr>
<tr>
<td>cmlara</td>
<td><span class="drupalorg-gitlab-issue-link drupalorg-gitlab-link-wrapper"><a href="https://git.drupalcode.org/project/gitlab_templates/-/issues/3343810" class="drupalorg-gitlab-link">https://git.drupalcode.org/project/gitlab_templates/-/issues/3343810</a></span> existed</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>doh</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>I'll clean up my dupe</td>
</tr>
<tr>
<td>cmlara</td>
<td>I kind-of asked this before:Is there a reason DrupalCi couldn't stay in parallel until November ?If not I question if effort should be spent on the D7 templates unless not having it will hold up other GitlabCi tasks.</td>
</tr>
<tr>
<td>Juraj Nemec</td>
<td>I am not suggesting to put more efforts into this instead of other D8+ Gitlab CI tasks. But probably need to have it in sight, because D7 EOL is not a fixed date - so probably this question will be more relevant around July 2023, when another decision should be published.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>DrupalCI will stay in parallel for a while, likely.</td>
</tr>
<tr>
<td>Gábor Hojtsy (he/him)</td>
<td>Yeah at this point we don't know if the November date is the final EOL or not, the way it was announced gives the possibility to extend it or not. Would know more latest in July. (edited)</td>
</tr>
</table>
<h2>@hestenet (he/him) I'm at DrupalSouth code sprint and @RoSk0 and @nick_schuch are looking at the core gitlab CI performance issue, can they ping you here with some questions?</h2>
<table>
<tr>
<td>larowlan</td>
<td>Nick was wondering about resource differences between jenkins and gitlab ci, e.g. cpu etc</td>
</tr>
<tr>
<td>larowlan</td>
<td>@hestenet (he/him) sorry in thread 🙂</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Let's see if I can pull up more precise details..</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>DrupalCI uses EC2 Spot Requests.The fleet type options are:r3.8xlarge, m4.10xlarge, c5.9xlarge, c3.8xlarge, c4.8xlarge</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>Hey @hestenet (he/him)!</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Hey @nick_schuch It's been quite a while! :smile:</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>The GitLab CI runner is configured on an EKS cluster.</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>It sure has ❤️</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>@nnewton is looking to rebuild/reconfigure the cluster before we get to DrupalCon - or at least update some things.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>@nnewton You had some input in our call this morning about how the scaling setup was not really configured right - can you speak to what sort of compute resources this is getting in its current config? I'm not sure I'm in the right spot in the EKS panel.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>IF - I'm reading this right, looking at the right cluster - it is also using:Instance typesr3.8xlargem4.10xlargec5.9xlargec3.8xlargec4.8xlarge</td>
</tr>
<tr>
<td>RoSk0</td>
<td>note: I've dropped Chromium service from kernel test suite to test and test test job went way further than previously when it timed out on 3 hoursrunning job <a href="https://git.drupalcode.org/project/gitlab_ci_testbed_for_drupal_core/-/jobs/47139previous">https://git.drupalcode.org/project/gitlab_ci_testbed_for_drupal_core/-/jobs/47139previous</a> one <a href="https://git.drupalcode.org/project/gitlab_ci_testbed_for_drupal_core/-/jobs/40955">https://git.drupalcode.org/project/gitlab_ci_testbed_for_drupal_core/-/jobs/40955</a></td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Well that's interesting...</td>
</tr>
<tr>
<td>RoSk0</td>
<td>@hestenet (he/him) it appears to me that all jobs are starting on the same runner - gitlab-prod-runner-gitlab-runner-66679bc9bd-dbkx6 JTJnATXCso, shouldn't we see different runners picking up the jobs if we have more than one? Or I'm misreading comment above ?</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>There's some server side config to get web terminals up for debugging that we just haven't been able to do (<a href="https://docs.gitlab.com/ee/ci/interactive_web_terminal/#configuration">https://docs.gitlab.com/ee/ci/interactive_web_terminal/#configuration</a>)</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>@RoSk0 Right now, this EKS cluster is not properly configured for resource groups for more runners. That's something we're working to fix. It didn't really need more than one runner with such a small number of folks testing it. That's a work in progress for sure.</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>(I have been typing for a while, I hope this is still relevant 🙂 )We have been debugging some of the performance issues on this issue.[#3325225]Steps taken:Compared the DrupalCI builds vs Gitlab CI buildsConfiguration looked very similar, tests were passing so I moved on to checking resource contention.Main difference I could see was related to DrupalCI builds running the steps sequentially vs GitLab CI fanning out and running kernel, units etc tests in parallel.Disabled All Other Tests Expect for KernelWe then proceeded to disable all the test suites except for one to see if the parallel testing was causing any issues.We picked kernel tests since it was running in 5(ish) minutes on DrupalCI and timing out after 3hrs on Gitlab CI.After disabling all except for Kernel we are now seeing it get alot further in it's test execution, it's still slow but feels like enough evidence for us to keep debugging in this direction.CPU ContraintsTaking a bit of a guess here, but this reminds me past issues I've had on our Kubernetes clusters were applications were given a really low CPU "limit" contraint in the Kubernetes Pod definition.That has led me to this chat :smile:Any chance we can see what the Pod definition looks like?</td>
</tr>
<tr>
<td>nnewton</td>
<td>@hestenet (he/him) our discussion this morning prolly is not super relevant here</td>
</tr>
<tr>
<td>nnewton</td>
<td>There are no limits, no</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Not once it finds one of the limited bots and starts running anyway? Okay.</td>
</tr>
<tr>
<td>nnewton</td>
<td>but there are also no requests, so there are currently no CPU guarantees at all and everything is running on a single spot instance</td>
</tr>
<tr>
<td>nnewton</td>
<td>so I am not sure what is being dbugged here, but if performance is generally "weird" or feels like contention. I am sure there is</td>
</tr>
<tr>
<td>nnewton</td>
<td>these jobs are basically all thrown with no definition for resources at the autoscaling asg, which creates a single node and runs them</td>
</tr>
<tr>
<td>nnewton</td>
<td>Which is what @hestenet (he/him) and I were discussing this morning, that they need resource requests</td>
</tr>
<tr>
<td>nnewton</td>
<td>so it could be related; depending on what is being debugged here</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>@nnewton Is it worth sharing the pod definition that @nick_schuch asked for in its current state?</td>
</tr>
<tr>
<td>nnewton</td>
<td>There is no pod definition</td>
</tr>
<tr>
<td>nnewton</td>
<td>gitlab-runner manages that</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Oh gotcha</td>
</tr>
<tr>
<td>RoSk0</td>
<td>I my experience I saw Docker executor runners was performing better than Kubernetes executor. Do any of you have similar experience?</td>
</tr>
<tr>
<td>nick_schuch</td>
<td><a href="https://docs.gitlab.com/runner/executors/kubernetes.htmlIs">https://docs.gitlab.com/runner/executors/kubernetes.htmlIs</a> cpu_limit set of the executor atm?</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>Happy to pause on it if this isn't being helpful</td>
</tr>
<tr>
<td>nnewton</td>
<td>Limits are not set. Honestly ya, performance eval is probably not particularly useful atm. I was just brought into this recently; as this cluster and runner setup was basically a PoC. It is almost unmanaged and very close to un-configured. Basically just configured enough to "work". Every job runs on the same node, never more than 1 can be spun up, because there are no resource requests. So, I'd imagine general performance is pretty bad.</td>
</tr>
<tr>
<td>nnewton</td>
<td>this week I re-configured the deployment to be actually managed/deployed well. Next step is setting some sane resource requests so that jobs run on more than one node.</td>
</tr>
<tr>
<td>nnewton</td>
<td>I'd be interested in what performance differences you are seeing, as I am still unsure what those resource reqs should be</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>Builds timing out after 3hrs (With all test suites running in parrallel)<a href="https://git.drupalcode.org/project/gitlab_ci_testbed_for_drupal_core/-/pipelines/10268Running">https://git.drupalcode.org/project/gitlab_ci_testbed_for_drupal_core/-/pipelines/10268Running</a> with a single test suiteIt was timing out after 3hrs with all the other tests running in parallel.Singling it out it now runs in 30min. Still failing, but looks to be unrelated.It was 5min on DrupalCI.<a href="https://git.drupalcode.org/project/gitlab_ci_testbed_for_drupal_core/-/pipelines/11535">https://git.drupalcode.org/project/gitlab_ci_testbed_for_drupal_core/-/pipelines/11535</a></td>
</tr>
<tr>
<td>nnewton</td>
<td>Also, do you have a comparison between these same jobs somewhere else. The jenkins jobs are sorta crazy. Running mysql entirely in tmpfs/etc</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>Any chance we could do a kubectl get pods on the cluster while I hit the rebuild button? :smile:</td>
</tr>
<tr>
<td>nnewton</td>
<td>sure?</td>
</tr>
<tr>
<td>nnewton</td>
<td>what are you looking for</td>
</tr>
<tr>
<td>nnewton</td>
<td>it has tokens in that definition, so I am not going to send the whole yaml</td>
</tr>
<tr>
<td>nnewton</td>
<td>Do you have a baseline runtime for these jobs?</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>Only going off the DrupalCI and Gitlab CI comparison atm</td>
</tr>
<tr>
<td>nnewton</td>
<td>k, that may not be a good comparison</td>
</tr>
<tr>
<td>nnewton</td>
<td>the drupalci jobs are nuts</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>for sure</td>
</tr>
<tr>
<td>nnewton</td>
<td>I was involved in making them not take hours way way back wehn</td>
</tr>
<tr>
<td>nnewton</td>
<td>when*</td>
</tr>
<tr>
<td>nnewton</td>
<td>mysql in tmpfs, no doublewrite, etc etc</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>I was looking to see if anything was being set on the Pod by default resourcing wise</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>and if that looks all good then we know the next steps are eks cluster related :+1: (edited)</td>
</tr>
<tr>
<td>nnewton</td>
<td>Maybe, it could also just be the jobs are slow</td>
</tr>
<tr>
<td>nnewton</td>
<td>This EKS cluster is a nightmare atm, but the jobs may just be slow if they've never been run anywhere else / we don't have a baseline</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Another potential avenue to explore if you want a runner environment you can more tightly control - is you should have permission to register your own runners on the test project: <a href="https://git.drupalcode.org/project/gitlab_ci_testbed_for_drupal_core/-/settings/ci_cdAnd">https://git.drupalcode.org/project/gitlab_ci_testbed_for_drupal_core/-/settings/ci_cdAnd</a> then anything you learn from that config (such as resource limits) we could pass back into the config for our shared runner as we update it.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>That might be one way to try and get some sort of 'baseline' as it were.</td>
</tr>
<tr>
<td>nnewton</td>
<td>Or running them outside of CI if possible. I've found it extremely hard to do this only in the CI pipeline</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>Absolutely, how about:I'll get the test rerun so I can quickly check out the Pod definition that gets generatedLook into runners so we can experiment</td>
</tr>
<tr>
<td>nnewton</td>
<td>sure, I will need to edit out quite a bit because I think there are tokens in the definition, but go for it</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>Starting to run. Waiting for the Kernel tests to kick off :+1:</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>:eyes: <a href="https://git.drupalcode.org/project/gitlab_ci_testbed_for_drupal_core/-/pipelines/11537">https://git.drupalcode.org/project/gitlab_ci_testbed_for_drupal_core/-/pipelines/11537</a> :eyes:</td>
</tr>
<tr>
<td>nnewton</td>
<td>Edited out some stuff but lots still here</td>
</tr>
<tr>
<td>nnewton</td>
<td>and you can see the empty resources block</td>
</tr>
<tr>
<td>nnewton</td>
<td>Ya and these are scheduled on a single node, because no requests. And that node has a load of 50</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>thanks @nnewton!</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>it's also running all the linting in parallel as well! :smile:</td>
</tr>
<tr>
<td>nnewton</td>
<td>Note: IMO these core tests should not mix with contrib</td>
</tr>
<tr>
<td>nnewton</td>
<td>and should be on their own runner with a much larger resource request</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>absolutely</td>
</tr>
<tr>
<td>nnewton</td>
<td>so any info you glean as to what the resource req ends up being, would be very useful to me</td>
</tr>
<tr>
<td>nnewton</td>
<td>as I'd love to not default to "everyone gets their own full node" but that is where I'm at atm.</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>definitely need some requests/limits and then have them as contrib vs core (edited)</td>
</tr>
<tr>
<td>nnewton</td>
<td>and/or different runners but use AWS Karpenter to spin up right-sized nodes for cost reduction. Either way...at some point I need to know how big these need to be heh</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>Fargate could also be an option for the long running core side of things</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>Price permitting</td>
</tr>
<tr>
<td>nnewton</td>
<td>Doesn't go big enough</td>
</tr>
<tr>
<td>nnewton</td>
<td>If these end up being like drupalci</td>
</tr>
<tr>
<td>nnewton</td>
<td>which, I imagine they will</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>what are the current sizes?</td>
</tr>
<tr>
<td>nnewton</td>
<td>the management of this cluster long term isn't really an issue once its in our pulumi trees. It just predates them and historically I wasn't dealing with this</td>
</tr>
<tr>
<td>nnewton</td>
<td>I think tim noted them above, but very large. As far as I know fargate goes to 16 vcpus and drupalci uses quite a bit larger and when we looked at fargate it wasn't really an option for that. Also, not sure it works/gets us much over spot fleets</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>oh! sorry Tim!</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>is it one core test per node?</td>
</tr>
<tr>
<td>nnewton</td>
<td>on drupalci? yes I believe so</td>
</tr>
<tr>
<td>nnewton</td>
<td>(this is from deep memory though heh)</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>:smile:</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>What is the size of the Gitlab CI node?</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>That's above as well one sec...</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td><a href="https://drupal.slack.com/archives/CGKLP028K/p1684450793121459?thread_ts=1684450210.332539&cid=CGKLP028K">https://drupal.slack.com/archives/CGKLP028K/p1684450793121459?thread_ts=1684450210.332539&cid=CGKLP028K</a></td>
</tr>
<tr>
<td>nick_schuch</td>
<td>Those are pretty beefy!</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Yeah.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>I think the trick may be figuring out how to replicate the trickery we did on DrupalCI to get these things to run faster.... or figure some other way to speed the tests up.</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>Which one of those node types is running atm @nnewton?</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>or Tim!</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>sorry for rehashing :smile:</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>Linting was also running in parrallel of the kernel tests</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>So maybe we also need to restructure the gitlab config to run everything sequentially like DrupalCI @RoSk0Then we can compare the difference and sort out that trickery :smile:</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>I think it was a c4.8xlarge - but amazon continues to be the worst ui ever to navigate.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>We're running over end of day here now - but I'll keep an eye on the thread.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>But yeah - I think self-registering a runner would help you control a lot of potential variables as well, for sure.</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>And wouldn't hurt to try the sequential thing, just to see...</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>Thankyou @hestenet (he/him) and @nnewton!!!!!!</td>
</tr>
<tr>
<td>RoSk0</td>
<td>yes, thanks all</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Thanks for looking at it! As Narayan said, we do need some guidance to figure out what the resource requests should actually be set to...</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>Is there anywhere I can look for the DrupalCI config trickery?</td>
</tr>
<tr>
<td>nick_schuch</td>
<td><a href="https://stackoverflow.com/questions/45081754/gitlabci-slow-build-with-docker-and-mysql-service">https://stackoverflow.com/questions/45081754/gitlabci-slow-build-with-docker-and-mysql-service</a></td>
</tr>
<tr>
<td>nick_schuch</td>
<td>This looks promising, time for a byo runner :smile: (edited)</td>
</tr>
<tr>
<td>larowlan</td>
<td>@xurizaemon</td>
</tr>
<tr>
<td>nick_schuch</td>
<td><a href="https://drupal.slack.com/archives/CGKLP028K/p1684452607456779?thread_ts=1684450210.332539&cid=CGKLP028K404">https://drupal.slack.com/archives/CGKLP028K/p1684452607456779?thread_ts=1684450210.332539&cid=CGKLP028K404</a> :disappointed:</td>
</tr>
<tr>
<td>RoSk0</td>
<td>are you saying you have no access @nick_schuch? (edited)</td>
</tr>
<tr>
<td>xurizaemon</td>
<td>Invite sent @nick_schuch</td>
</tr>
<tr>
<td>xurizaemon</td>
<td>First things I suggest to try are a BYO runner to get a comparative measure of run time outside that cluster</td>
</tr>
<tr>
<td>xurizaemon</td>
<td>Lee suggested last night (at the pub, forgive inaccuracy!) that MySQL was placing data in /dev/shm ... I'd also consider dropping that to get a baseline of MySQL on default storage. Our experience with Chrome using /dev/shm in CI was that it behaved much better when that was disabled entirely. My impression is that shared memory in runners is not reliable (there may be contention between runners, I have no idea whether treating it as a volume mount with concurrent runners is a good idea).</td>
</tr>
<tr>
<td>xurizaemon</td>
<td>(Guessing the runners don't have spinning disc storage anyway, but IDK)</td>
</tr>
<tr>
<td>xurizaemon</td>
<td>I didn't find yet where any configuration of the MySQL service is placed (because I did not look yet).</td>
</tr>
<tr>
<td>nick_schuch</td>
<td>Thanks @xurizaemon I'll get my digitalocean (60 day free credit 🙂 ) runner up and going. So far it looks like the difference for Kernel tests is 5min on the old bot vs 30min on the new.I plan on put time in this week to determine if the in memory db approach is the major reason for this 25min difference.</td>
</tr>
<tr>
<td>xurizaemon</td>
<td>Nice one, thanks @nick_schuch. Happy to pair up if you ping me when looking and I am able - might be good? (edited)</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>Heads up - the GitLabCI cluster now has a pretty beefy resource request in place - each pipeline should be given a 30 vcpu request - ensuring basically 1 whole machine per job.That should eliminate some of the possible confounding variables on testing performance. We may be able to make more progress at DrupalCon. (edited)</td>
</tr>
<tr>
<td>hestenet (he/him)</td>
<td>3️⃣ 6️⃣ (reserved)</td>
</tr>
</table>
issue