fix: #3636849 Fix five bugs that can give a broken or incomplete archive

Thomas Alsénrequested to merge
issue/earkiv-3636849:3636849-fix-five-bugs into 1.0.x

Closes https://www.drupal.org/project/earkiv/issues/3636849

One commit per bug, so they can be reviewed one at a time.

Commits

  1. Fail the job when the crawl fetched nothing. HTTrack exits with 0 also when the host cannot be reached or the start page returns an error. checkCrawl() now reads the URL log and fails the job (with the usual resume attempts) if no URL returned a successful response or the start URL returned an error.
  2. Stop a leftover HTTrack before the crawl is continued. stopHttrack() finds HTTrack processes running with --path in the working directory of the job (via ps), sends SIGTERM and after 10 s SIGKILL, and removes the HTTrack lock file. It runs before the crawl is started or continued, and when a job is cancelled.
  3. Remove the archive files however an archive is deleted. The overridden Archive::delete() is replaced by Archive::preDelete(), which lets the archiver and storage plugins remove their working files (via their cancel()) and deletes the archive file. drush entity:delete and bulk deletion now remove the file too. When a job fails for good, its working directory is removed; the archive file, if any, is kept.
  4. Let the plugins process their settings in the forms. The settings page and the archive form now call submitConfigurationForm() of the plugins, like drush earkiv:create already did. A configuration schema for earkiv.settings is added, and earkiv_post_update_process_plugin_settings() processes the existing default settings.
  5. Replace HTML error pages saved under another extension. Whether an error page is HTML is decided from the MIME type in the URL log instead of the file extension. Such pages get the notice as <file>.html, and HTTrackMirror::replaceLinkTargets() points the links to it. The README is updated.

Testing

  • Unreachable host: "No page could be fetched from …"; start page with 404: "The URL … returned HTTP status 404". Both jobs end as Error after the resume attempts.
  • Killing only the Drush process during a crawl: both drush earkiv:process (resume) and drush earkiv:cancel stopped the leftover HTTrack.
  • drush entity:delete earkiv_archive removed the archive file, its file entity and the working directories; a failed job had its working directory removed.
  • Settings form and archive form with a storage directory /earkiv-test/, CRLF line breaks and the number of connections as a string saved earkiv-test, \n and an integer. The schema validates. The post update processed the existing settings of a site.
  • A small mirror with error pages saved as index.se, index.pdf and index.png: the first two became notices as .html with the links pointed to them (fragments kept), the PNG was removed as before.
  • Full archive of a Drupal 11 site with all fixes: 7,876 URLs, 110 broken links, 2.39 GB, 12 min 41 s with 128 connections. 16 error pages with another extension became notices, and the links to them work.
  • phpcs (Drupal, DrupalPractice) reports no new findings compared with 1.0.x.

stopHttrack() uses ps and only sees processes on the same server.

AI-Generated: Yes (Used Claude Code to find and fix the bugs, test them and write this merge request.)

Merge request reports