feat: #3636555 Add start URLs and sitemap.xml as crawl entry points

Thomas Alsénrequested to merge
issue/earkiv-3636555:3636555-add-start-urls into 1.0.x

Closes https://www.drupal.org/project/earkiv/issues/3636555

What this changes

  • Include URLs from sitemap.xml (new checkbox, off by default). Before the crawl, sitemap.xml at the archived host is read, sitemap indexes are followed (max 2 levels, max 1,000 sitemaps), and every listed URL becomes a start URL.
  • Additional start URLs (new textarea, one URL per line, validated as http or https URLs).
  • Start URLs are written to start-urls.txt in the working directory, outside the mirror so it is not part of the archive, and passed to HTTrack with -%L. They are also used when a crawl is continued.
  • Host rewrite. Sitemaps often list the production host also on test and local environments (Simple XML Sitemap uses its base URL setting, and the sitemap index points to https://www.example.com/sitemap.xml?page=N). Sitemap and page URLs are therefore moved to the scheme and host of the archived URL, keeping path and query, before they are fetched or crawled. Without this, archiving a test site would crawl production.
  • Exclude patterns apply to start URLs. HTTrack always fetches the URLs it is given, so start URLs matching an "Exclude URLs" pattern are filtered out first, matched the same way as HTTrack filters (* as wildcard, URL with or without scheme).
  • A sitemap that cannot be read or parsed is logged as a warning and the crawl continues from the archived URL only.

Testing

Tested in DDEV on a Drupal 11 site with 1,984 published nodes.

  • Reading the sitemap index and its 9 sitemaps took 0.4 s and gave 1,696 unique start URLs, all moved from the production host to the DDEV host. An exclude pattern removed the matching start URL.
  • A small job with three additional start URLs outside the start directory archived all three pages. start-urls.txt is not in the archive.
  • Full archive from the site root, sitemap on, 8 connections, same exclude patterns as before plus two for duplicate order forms:
Before (sitemap off) After (sitemap on)
Time, crawl + archive + storage 78 min 30 s 83 min 41 s
Archive size 2.41 GB 2.44 GB
HTML pages 3,813 4,268
Published nodes missing 417 135

All 264 remittance pages are now archived. The 135 still missing are contact persons (103) and press images (20), which are not in the sitemap, plus 12 pages that are neither linked nor in the sitemap.

Notes for reviewers

  • When the archived URL is a subdirectory, HTTrack stays in that directory by default, so documents and images linked from additional start URLs elsewhere are not fetched. Archiving from the site root, which is the normal case, is not affected.
  • The new $httpClient property has no doc comment, like the existing properties in the same class.

AI-Generated: Yes (Used Claude Code to write the code changes, run the tests in DDEV and draft this description.)

Merge request reports