feat: #3636718 Treat alternative host names as the same site

Thomas Alsénrequested to merge
issue/earkiv-3636718:3636718-treat-alternative-host into 1.0.x

Closes https://www.drupal.org/project/earkiv/issues/3636718

What it does

Adds the HTTrack setting Alternative host names (and --alias-host for drush earkiv:create): other host names of the same site, one per line. After the crawl:

  1. Absolute href/src links in the archived HTML pages that point to an alternative host are moved to the archived host. URLs that match an exclude pattern are left out.
  2. Those not yet in the archive are fetched in a second, list-only HTTrack run (-r1), which is merged into the mirror together with its URL log. Log lines without a referring page get the page that contains the link, so broken links from the second run end up in earkiv-broken-links.csv with a source page.
  3. After the error pages are handled, links to the alternative hosts and absolute links to the archived host are rewritten to relative links when the target is in the archive. Anything else is left as it is.

HTTrack does not save error pages for URLs from a list (nor for some URLs with special characters), so a notice page is now also created when the error page file is missing. That way links to missing pages lead to the notice in the archive instead of the live site.

The new class HTTrackMirror reads hts-cache/new.txt to map URLs to local files, finds and rewrites links, and merges mirrors.

Testing

Full archive of a Drupal 11 site (about 7,500 URLs) made from a local copy, with the production host names www.example.se and example.se as alternative host names:

Without With
Links to the production host 1,145 1 (a URL with a trailing period in the content)
Linked PDF only reachable via full production URL Not archived Archived, relative link
Broken links in the CSV 72 109 (links to the production host that lead to missing pages, previously not checked)
Time 39 min 20 s 40 min 15 s

The second run fetched 158 files for 195 links. Handling error pages and rewriting links took about 9 seconds for the whole site. phpcs (Drupal, DrupalPractice) reports no new findings compared with 1.0.x.

AI-Generated: Yes (Used Claude Code to write the code, test it on a full archive and write this merge request.)

Merge request reports