feat: #3636813 Archive published pages that no other page links to

Thomas Alsénrequested to merge
issue/earkiv-3636813:3636813--archive-published into 1.0.x

Closes https://www.drupal.org/project/earkiv/issues/3636813

What it does

Adds the HTTrack setting Include all published content (and --published-content for drush earkiv:create). When the crawl starts, every published node that the anonymous user may view is added to the start URLs, in every language, with the path alias and language prefix from toUrl(). The URLs are moved to the archived host and URL-encoded (path aliases can contain +, which Drupal answers with 404 unless it is encoded).

  • Rabbit Hole: if it is enabled and handles nodes, only nodes whose action is display_page are included. Both the bundle setting and the override on the node are respected, and hook_rabbit_hole_values_alter() is invoked. Rabbit Hole 1.x is read through rabbit_hole.behavior_invoker, 2.x through rabbit_hole.behavior_settings_manager::getEntityBehaviorSettings(). Rabbit Hole stays optional.
  • The nodes set as the 403 and 404 pages in system.site are left out, as are URLs that match an exclude pattern.
  • earkiv-unlinked-pages.csv lists the published pages that no other page in the archive links to (nid, language, type, title, URL, file). Only links in a and area elements count. Links between translations of the same node, and from the copy of the page HTTrack saves when it follows the /node/NID shortlink, do not count. The front page is not listed.
  • Published pages that return an error and that no page links to are not listed as broken links.

The new service earkiv.published_content (PublishedContent) lists the pages; HTTrackMirror::unlinkedFiles() finds the files no page links to.

Testing

Full archive of a Drupal 11 site with 1,984 published nodes and Rabbit Hole 1.3:

  • 1,959 published pages were added as start URLs (without Rabbit Hole: 1,982; the difference is 22 nodes Rabbit Hole answers with "Page not found" and 1 that it redirects). Listing them took 1.4 s.
  • Pages that were missing before, such as thank-you and result pages, are now in the archive. 7,877 URLs instead of 7,557, 2.39 GB, 42 min (40 min before).
  • 2 published pages returned 404 and were left out of the broken links. That run found that + in path aliases had to be encoded, which is fixed.
  • The unlinked page report, rerun on that archive with the final code, lists 387 pages, among them 261 consultation pages whose listing links straight to the PDF files instead of the pages, and 103 contact persons. Finding them took about 1.3 s.

phpcs (Drupal, DrupalPractice) reports no new findings compared with 1.0.x. Rabbit Hole 2.x support follows the 2.0.0-beta3 API but has not been tested on a site with 2.x.

AI-Generated: Yes (Used Claude Code to write the code, test it on a full archive and write this merge request.)

Merge request reports