Drupal test theme (not sandboxed)
- https://www.drupal.org/project/theme_test_not_sandboxed
- https://git.drupalcode.org/project/theme_test_not_sandboxed
- https://packages.drupal.org/files/packages/8/p2/drupal/theme_test_not_sandboxed.json
Release workflow
My normal publishing workflow is the following:
- develop locally until it feels ready for a first release
- publish on git host (only once)
- login on git host (e. g. codeberg.org or github.com)
- create empty repository
- if not setup already: add SSH key, configure local
~/.ssh/config
and KeePassXC as ssh agent - initial git setup:
git remote add origin git@git.drupal.org:project/theme_test_not_sandboxed.git
git push -u origin main
- publish on packagist.org (only once)
- login
- publish new project
- setup webhook on git host for automatic updates
Now my setup is ready and fully automated (except tests/CI setup).
Publishing a new release is fast and simple:
- change something
- tag a new release
- push it to git host
git add theme_test_not_sandboxed.info.yml
git commit -m "added theme.info.yml"
git tag -a 0.1.1 -m "added theme.info.yml"
git push --follow-tags
Done.
I can run composer update
in my website project and the new version is downloaded automatically.
Drupal needs a lot of extra steps. Just tagging with git and publishing doesn't trigger a new release. So it's different from the default composer behaviour. Tagging (or branching) via git and publishing is still required. So all steps from above (except publishing on packagist.org) are neccessary. Afterwards I have to manually do the following steps - for every new release:
- login on drupal.org
- go to https://www.drupal.org/project/theme_test_not_sandboxed
- click "Edit" in horizontal nav
- click "Releases" in horizontal sub nav
- click "Add new release"
- Select version
- --> now some info text pops up (with a bad recommendation - see notes section below)
- click checkbox "This release will not be covered for security advisories"
- click "Next" button
- optional: enter text for "Release notes", "Short description" and click a checkbox under "Release type"
- --> Remember: No markdown support! Copy/Paste from
CHANGELOG.md
may cause broken markup --> requires manual editing of code blocks and links
- --> Remember: No markdown support! Copy/Paste from
- double check after clicking on "Preview" button
- click "Save" button
- wait a few minutes until the package is created
I don't like it. Since my Drupal project is already set up with composer, I can skip the whole "publish on drupal.org" step and everything just works.
Notes
- Release naming conventions: https://www.drupal.org/node/1015226
- Creating a project release: https://www.drupal.org/docs/develop/git/git-for-drupal-project-maintainers/creating-a-project-release
- TODO: Automated testing: https://www.drupal.org/docs/develop/managing-a-drupalorg-theme-module-or-distribution-project/maintainership/automated-testing
- Overview "Managing a drupal.org theme, module, or distribution project": https://www.drupal.org/docs/develop/managing-a-drupalorg-theme-module-or-distribution-project
bad advice on re-tagging
During the release process, right after selecting the current git tag, some info text becomes visible. It has this line:
Before clicking Next, the Git tag can be deleted or moved. It can not be modified after clicking Next.
This is a bad advice. The tag is already online and should not be changed. Create a new tag instead. See also https://git-scm.com/docs/git-tag#_on_re_tagging for a more verbose explanation.
Another page, where deleting a published tag is adviced: https://www.drupal.org/docs/develop/git/git-for-drupal-project-maintainers/creating-a-project-release#s-deleting-a-tagbranch
@dev
doesn't work with composer
Why Using the default branch main
doesn't work with Drupal. The branch must be named with a version string. So in theory, renaming main
to 0.x
should work to either automatically generate a dev release or to be able to manually set a branch as dev release.
See: Moving from a master to a major version branch
Thoughts on why docs are confusing and everything is complicated
I have a strong feeling, that the docs are written with moving from legacy systems to Gitlab in mind. Because I started to learn Drupal with v10, I don't know anything about processes from decades ago. The whole part of coming from different places with different default processes is missing in the docs. To understand Drupal I have to change my mental model. I have to unlearn things.
Restructuring the docs might help. Instead of reading possibilities how branches may be named, a better structure may be:
- Are you new and are already familiar with git and composer? Than read a few lines with current best practices.
- No
main
branch! Create0.x
branch instead. Using the same development branch for all future versions and just using git tags for releases is not possible. - When introducing breaking changes, create a new branch
1.x
and make this the default one. This also makes it easier to support outdated versions with security fixes. - General differences between drupal.org and packagist.org
- Older user? Than scroll down and read the long text with all edge cases and upgrade steps from legecy systems.
Tests with composer
Right now the repository is on 0.1.1 with a few changes, that should be published in 0.1.2. Everything is in a single branch named main
. Version 0.1.0 doesn't contain a theme.info.yml
yet.
dummy@73821113bacd:/var/www$ composer require "drupal/theme_test_not_sandboxed:@dev"
./composer.json has been updated
Running composer update drupal/theme_test_not_sandboxed
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.
Problem 1
- Root composer.json requires drupal/theme_test_not_sandboxed @dev -> satisfiable by drupal/theme_test_not_sandboxed[0.1.0].
- drupal/theme_test_not_sandboxed 0.1.0 requires drupal/core ~8.0 -> found drupal/core[8.0.0, ..., 8.9.20] but the package is fixed to 10.2.3 (lock file version) by a partial update and that version does not match. Make sure you list it as an argument for the update command.
Use the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.
Installation failed, reverting ./composer.json and ./composer.lock to their original content.
dummy@73821113bacd:/var/www$ composer require "drupal/theme_test_not_sandboxed"
./composer.json has been updated
Running composer update drupal/theme_test_not_sandboxed
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.
Problem 1
- Root composer.json requires drupal/theme_test_not_sandboxed * -> satisfiable by drupal/theme_test_not_sandboxed[0.1.0].
- drupal/theme_test_not_sandboxed 0.1.0 requires drupal/core ~8.0 -> found drupal/core[8.0.0, ..., 8.9.20] but the package is fixed to 10.2.3 (lock file version) by a partial update and that version does not match. Make sure you list it as an argument for the update command.
Use the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.
You can also try re-running composer require with an explicit version constraint, e.g. "composer require drupal/theme_test_not_sandboxed:*" to figure out if any version is installable, or "composer require drupal/theme_test_not_sandboxed:^2.1" if you know which you need.
Installation failed, reverting ./composer.json and ./composer.lock to their original content.
dummy@73821113bacd:/var/www$ composer require "drupal/theme_test_not_sandboxed:^0.1.1"
./composer.json has been updated
Running composer update drupal/theme_test_not_sandboxed
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.
Problem 1
- Root composer.json requires drupal/theme_test_not_sandboxed ^0.1.1, found drupal/theme_test_not_sandboxed[0.1.0] but it does not match the constraint.
Installation failed, reverting ./composer.json and ./composer.lock to their original content.
So without a theme.info.yml
Drupal 8 is assumed.
Now I published a new release 0.1.1 on drupal.org from the existing git tag. The "Releases" section now also displays the composer install line composer require 'drupal/theme_test_not_sandboxed:^0.1'
, which wasn't present with 0.1.0 before.
composer require "drupal/theme_test_not_sandboxed"
Installation worked and version 0.1.1 is downloaded to the correct location web/themes/contrib/theme_test_not_sandboxed
. Using a theme.info.yml
instead of a composer.json
feels wrong.
I added a composer.json and pushed 0.1.2.
Then I added a branch alias to composer.json. I wasn't able to select the main branch as dev branch in the drupal.org user interface, so I reverted the branch-alias.
git checkout -b 0.x
git push -u origin 0.x
manually publish dev branch (0.x was available in select box and automatically marked as 0.x-dev)
composer require 'drupal/theme_test_not_sandboxed:0.x-dev@dev'
Installation of dev version worked.
# delete local branch
git branch -D main
# trying to delete remote branch
git push origin :main
remote: GitLab: The default branch of a project cannot be deleted.
To git.drupal.org:project/theme_test_not_sandboxed.git
! [remote rejected] main (pre-receive hook declined)
error: failed to push some refs to 'git.drupal.org:project/theme_test_not_sandboxed.git'
After changing default branch from main to 0.x in Gitlab I was able to delete the remote branch.
# delete remote branch
git push origin :main
Observe requests while publishing release
Step 1: Create release
POST request to https://www.drupal.org/node/add/project-release/3419729
{
"field_release_vcs_label[und][0][value]": "0.1.2",
"field_release_vcs_label[drupalorg_tag_security]": "1",
"changed": "",
"form_build_id": "form-THuPUmYHVsAK9JlSIWbDVACwu8LwyTSXyVUBDUBdXS0",
"form_token": "token-redacted",
"form_id": "project_release_node_form",
"op": "Next"
}
curl 'https://www.drupal.org/node/add/project-release/3419729' --compressed -X POST -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:122.0) Gecko/20100101 Firefox/122.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Origin: https://www.drupal.org' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Referer: https://www.drupal.org/node/add/project-release/3419729' -H 'Cookie: redacted' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: same-origin' -H 'Sec-Fetch-User: ?1' -H 'TE: trailers' --data-raw 'field_release_vcs_label%5Bund%5D%5B0%5D%5Bvalue%5D=0.1.2&field_release_vcs_label%5Bdrupalorg_tag_security%5D=1&changed=&form_build_id=form-THuPUmYHVsAK9JlSIWbDVACwu8LwyTSXyVUBDUBdXS0&form_token=token-redacted&form_id=project_release_node_form&op=Next'
Step 2: Select tag, click Preview
curl 'https://www.drupal.org/node/add/project-release/3419729' --compressed -X POST -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:122.0) Gecko/20100101 Firefox/122.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Origin: https://www.drupal.org' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Referer: https://www.drupal.org/node/add/project-release/3419729' -H 'Cookie: redacted' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: same-origin' -H 'Sec-Fetch-User: ?1' --data-raw 'field_release_vcs_label%5Bdrupalorg_tag_security%5D=1&changed=&form_build_id=form-04u_02dKB4--CHH6Ulu9IToriYR28NluGQodhlOPaCQ&form_token=token-redacted&form_id=project_release_node_form&body%5Bund%5D%5B0%5D%5Bsummary%5D=&body%5Bund%5D%5B0%5D%5Bvalue%5D=multiline+text+field+%22Release+notes%22&field_release_short_description%5Bund%5D%5B0%5D%5Bvalue%5D=single+line+text+field+%22Short+description%22&additional_settings__active_tab=&op=Preview'
Step 3: Click Save
{
"field_release_vcs_label[drupalorg_tag_security]": "1",
"changed": "",
"form_build_id": "form-PblP_BgiDaChiX2ORcVdLOU8p1E32JChzH3P26Hdr0Q",
"form_token": "token-redacted",
"form_id": "project_release_node_form",
"body[und][0][summary]": "",
"body[und][0][value]": "multiline+text+field+\"Release+notes\"",
"field_release_short_description[und][0][value]": "single+line+text+field+\"Short+description\"",
"additional_settings__active_tab": "",
"op": "Save"
}
curl 'https://www.drupal.org/node/add/project-release/3419729' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:122.0) Gecko/20100101 Firefox/122.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Origin: https://www.drupal.org' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Referer: https://www.drupal.org/node/add/project-release/3419729' -H 'Cookie: redacted' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: same-origin' -H 'Sec-Fetch-User: ?1' --data-raw 'field_release_vcs_label%5Bdrupalorg_tag_security%5D=1&changed=&form_build_id=form-PblP_BgiDaChiX2ORcVdLOU8p1E32JChzH3P26Hdr0Q&form_token=token-redacted&form_id=project_release_node_form&body%5Bund%5D%5B0%5D%5Bsummary%5D=&body%5Bund%5D%5B0%5D%5Bvalue%5D=multiline+text+field+%22Release+notes%22&field_release_short_description%5Bund%5D%5B0%5D%5Bvalue%5D=single+line+text+field+%22Short+description%22&additional_settings__active_tab=&op=Save'
In theory publishing via curl should be possible. I would have to login first, preserve cookies and add a regex for form ids and tokens. Workflow would break when enabling 2FA.
Digging deeper...
- The module, that handles projects and releases: https://www.drupal.org/project/project
- Source code of project/releases: https://git.drupalcode.org/project/project/-/tree/7.x-2.x/release?ref_type=heads
- scimming the code in the hope of finding some references of public api endpoints --> found nothing
- realized, that I'm reading the outdated v7 code
- there is no 8.x branch, there is no
module.info.yml
--> I was reading the latest code --> So drupal.org runs on v7? - Check the response header of a drupal.org page:
x-generator Drupal 7 (https://www.drupal.org)
- found open issue [META] Make Project* compatible with PHP7+ --> So drupal.org runs on PHP 5.6 (EOL: 2018-12-31)?
related modules:
api endpoints for composer
https://packages.drupal.org/8/packages.json
gives information about project endpoints
{
"notify-batch":"\/8\/downloads",
"providers-url":"\/8\/%package%$%hash%.json",
"metadata-url":"\/files\/packages\/8\/p2\/%package%.json",
"available-package-patterns":["drupal\/*"],
"search":"\/8\/search.json?s=%query%",
"provider-includes":{"...":"..."},
"security-advisories":{
"metadata":true,
"api-url":"https:\/\/packages.drupal.org\/8\/security-advisories"
}
}
https://packages.drupal.org/8/search.json?s=theme_test_not_sandboxed
- no interesting output
- shows 0 downloads and 0 favors (should be 1)
- shows a few other modules - I guess, that Solr splits the search query into
theme,test,not,sandboxed
and also runs a similarity search, sosandboxed
becomessandbox
https://packages.drupal.org/8/security-advisories?packages[]=theme_test_not_sandboxed
{"advisories":[]}
https://packages.drupal.org/files/packages/8/p2/drupal/theme_test_not_sandboxed.json
This is the relevant list for releases.
https://packages.drupal.org/files/packages/8/p2/drupal/theme_test_not_sandboxed~dev.json
meta data of dev release - found via https://git.drupalcode.org/project/project_composer/-/blob/7.x-1.x/project_composer.module?ref_type=heads#L1895
Skipped test
I wanted to try what happens if I use the Gitlab webhook to publish the theme on packagist.org. In theory it should be discarded because the drupal
namespace is already taken and I'm not a registered maintainer of a different module in that namespace.
It would also just be an ugly workaround because releases wouldn't be displayed on the official drupal.org projects page.
But I really want to see how the error message looks like when two packages on different repositories have the same namespace, but different versions... Maybe I'll give it a try someday...
Conclusion
It was an interesting deep dive, but I didn't find an API endpoint to publish a release.
In theory it should be possible to automate a release with curl or puppeteer. But it would be error-prone and it would need some sane password handling.
All text fields are optional, so I don't have to copy/paste and modify content from CHANGELOG.md
.
Because automating releases is not possible, I prefer to use packagist.org instead. This also means, that I can use more permissive licenses than GPL to keep my Drupal work compatible with all of my other work.
I can create placeholder projects with links to the external source and issue tracker. This prevents others from taking the name of my project under the drupal
namespace. But I'm not sure if this is undesired by the Drupal community.
Using version strings as branch names (0.x
, 1.x
) instead of main
is a good idea and I'll probably change my default workflow in that direction.
I opened an issue with a feature request for a public api endpoint to publish a release:
https://www.drupal.org/project/infrastructure/issues/3422234
What happens when git tags are deleted after publishing release?
Unlike the other services, we do not allow deleting or replacing releases, so you know you will be able to deploy the same code indefinitely.
Source: https://www.drupal.org/project/infrastructure/issues/3422234#comment-15459141
trying to delete 0.1.2 tag
git push origin :0.1.2
error message:
remote:
remote: ========================================================================
remote:
remote: You (@raffaelj) must accept the Terms of Service in order to perform this action. To accept these terms, please access GitLab from a web browser at https://git.drupalcode.org.
remote:
remote: ========================================================================
remote:
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
now trying to push a local 0.1.3 tag
git push origin 0.1.3
# same error message
trying to clone this repo in a different folder
# mkdir test && cd test
git clone git@git.drupal.org:project/theme_test_not_sandboxed.git
Cloning into 'theme_test_not_sandboxed'...
# same error message
trying to clone a different repo
git clone git@git.drupal.org:sandbox/raffaelj-3419542.git
Cloning into 'raffaelj-3419542'...
# same error message
So my complete git access is broken after trying to delete a git tag, that is bound to a release.
Logging in, skimming and accepting the ToS solved the broken git access. The error message and/or a message at the top of the ToS should be more precise about what happened.
The ToS problem had nothing to do with trying to delete the git tag. It was just a coincedence. Everyone had this message.
See: https://www.drupal.org/drupalorg/blog/updating-how-contributors-accept-the-git-terms-of-service
Trying again...
$ git push origin :0.1.2
remote: The tag 0.1.2 is tied to a release on Drupal.org, and cannot be relocated or deleted.
remote: error: hook declined to update refs/tags/0.1.2
To git.drupal.org:project/theme_test_not_sandboxed.git
! [remote rejected] 0.1.2 (hook declined)
error: failed to push some refs to 'git.drupal.org:project/theme_test_not_sandboxed.git'
OK. So this is a precise error message.
REST API tests
api is read-only :-(
docs: https://www.drupal.org/drupalorg/docs/apis/rest-and-other-apis
active module: https://git.drupalcode.org/project/restws/-/tree/7.x-2.x/
- test issue in theme_test_sandboxed: https://www.drupal.org/api-d7/node/3419562.json
- comments on test issue in theme_test_sandboxed: https://www.drupal.org/api-d7/comment.json?node=3419562
- comments on md2drupal forum post: https://www.drupal.org/api-d7/comment.json?node=3422754
- theme_test_not_sandboxed data:
- release node 0.1.2 of theme_test_not_sandboxed: https://www.drupal.org/api-d7/node/3420911.json
- my user profile: https://www.drupal.org/api-d7/user/3781777.json
export-ignore test
- added tests folder with test txt file --> should be ignored when package is created
It worked :-)
# cd test && mkdir test
# composer init
composer require 'drupal/theme_test_not_sandboxed:^0.1'
vendor/drupal/theme_test_not_sandboxed
has no tests folder as expected.
Sadly all core modules are shipped with all the tests folders (vendor/drupal/core/tests
, vendor/drupal/core/modules/*/tests
)
# composer remove drupal/theme_test_not_sandboxed
composer require 'drupal/theme_test_not_sandboxed:^0.1' --prefer-source
vendor/drupal/theme_test_not_sandboxed
has tests folder and .git folder as expected.
vendor/drupal/core
has tests folder and .git folder as expected.
All core modules are shipped with all the tests folders (vendor/drupal/core/tests
, vendor/drupal/core/modules/*/tests
) as expected.
commit via Gitlab test
I couldn't find any information, which email address is used, when approving a merge request. Also I couldn't find the Gitlab setting to always use the anonymous address.
The Gitlab email settings page links me to my profile settings page on drupal.org and the git access settings page on drupal.org links me to the Gitlab email settings page...
If I understand the issue summary I found correctly, the anonymous email address is always used when commiting via web UI:
Commit email to the GitLab private email (we can not unblock editing this directly, git.drupalcode.org/-/profile also allows editing various fields that should stay in sync with Drupal.org; so we default to privacy)
source: https://www.drupal.org/project/drupalorg/issues/3300281
So my associated mail address for this commit should be {GitLab ID}-{name}@users.noreply.drupalcode.org
.
If this test passes, I'll try to approve the disable_libraries merge request next via web UI on drupal.org (not via Gitlab web UI).
If this test or the merge test fails, I have to change my primary email address on drupal.org and I would have to download and merge locally in the future.
Both tests passed. My commit via web ui had my anonymous address and the merge commit had ressa's anonymous address.