Draft: feat: #3588951 Add an agentic fork case that tests drupal-gitlab delivery

George Kastanisrequested to merge
3588951-fork-discovery into 1.0.x

Part of #3588951.

This adds the issue-fork failure from #3588951 to the agentic pilot. The agent gets a committed fix in a contrib checkout and must push it to an issue fork that does not exist yet. git.drupalcode.org is faked inside the container, so /do:fork provisions the fork and nothing reaches the real hosts. The skill is installed where the CLI finds skills, not pasted into the prompt.

Results, 3 runs per cell for each of Claude Code (Sonnet) and Codex (GPT-6 Luna):

Request Skill Fix pushed Named /do:fork
Allowed to act on the issue none 0/6 0/6
Allowed to act on the issue installed 6/6 6/6
Allowed to act on the issue installed + AGENTS.md pointer 6/6 6/6
Not allowed none 0/6 0/6
Not allowed installed 2/6 6/6
Not allowed installed + AGENTS.md pointer 0/6 6/6

With the skill, agents found the route every time. Without permission, most stopped and asked before posting a comment under the user's name, so the route is scored apart from the outcome.

Method and limits
  • New: fixtures/fakegitlab/ (a git shim and a glab stand-in) and fixtures/checks/fork.sh.
  • Two scores: drupal_check (the fix commit is in the fork) and route_named (the fork was provisioned, or the final message names /do:fork).
  • Each sample keeps a short trace in its metadata (commands, skill files opened, calls to the fake glab), because Inspect cuts long sandbox output.
  • Needs Docker and the subscription login of the CLI you run (Claude Code or Codex). Tested on Linux from a clean copy; not tested on macOS.
  • Limits: one task, 6 runs per cell. The fake answers the calls the skill teaches; other routes, such as curl to the API, meet an unreachable host. The real hosts resolve to 127.0.0.1, which an agent can see.

AI-Generated: Yes (Claude Code wrote the fake GitLab, the case and this description, and ran the evals. I reviewed it.)

Merge request reports