feat: #3636583 Replace error pages in the archive with a notice and list broken links

Thomas Alsénrequested to merge
issue/earkiv-3636583:3636583-replace-error-pages into 1.0.x

Closes https://www.drupal.org/project/earkiv/issues/3636583

What this changes

After a successful crawl, before the archive is created, hts-cache/new.txt is read and every URL that returned HTTP 400 or higher, or an HTTrack connection error (negative code), is handled:

  • Saved as HTML: the file is replaced with a short notice page: the page did not exist when the site was archived, the HTTP status, the original URL and the date. Links in the archive keep pointing to it, so they lead to the notice instead of the site 404 page with broken menus.
  • Saved as another file type: the file is deleted, since it contains the HTML error page under a wrong extension.
  • All broken URLs are written to earkiv-broken-links.csv in the root of the archive: status, url, linked_from.

A URL logged more than once (for example after a continued crawl) uses its last status. The local file name is looked up both as logged and URL-decoded, because HTTrack logs it URL-encoded when the URL contains spaces or non-ASCII characters while the file on disk is decoded. Files outside the mirror directory are never touched.

The notice text is translatable and uses the default language of the site.

Testing

Tested in DDEV on a Drupal 11 site.

  • Mirror of a page with four broken links plus a fake missing PDF: all five listed in the CSV, the HTML error pages replaced and still linked from the page, the fake PDF removed.
  • Job through drush earkiv:process from a page with two broken links: both listed in earkiv-broken-links.csv in the archive, job ended as Done.
  • Full archive of the site (7,179 files), unpacked at the original crawl path: 72 broken URLs listed, all 56 HTML error pages replaced, no copy of the site 404 page left. Took under a second.

Notes for reviewers

  • 16 of the 72 broken URLs were not saved as HTML, mostly e-mail addresses entered as page links (for example info@example.com saved as index.com). Those files are removed, so the link in the archive leads to a missing file; the URL is still listed in the CSV.
  • Running HTTrack with -o0 (do not generate error pages) was considered, but HTTrack still rewrites the links to the local file, which then does not exist.

AI-Generated: Yes (Used Claude Code to write the code changes, run the tests in DDEV and draft this description.)

Merge request reports