fix: #3636404 Make archiving faster and the archive complete

Thomas Alsénrequested to merge
issue/earkiv-3636404:3636404-archiving-is-slow into 1.0.x

Closes https://www.drupal.org/project/earkiv/issues/3636404

What this changes

  • Faster crawl. HTTrack runs with --check-type=0 -%N0 --disable-security-limits, a configurable number of connections (--sockets, default 8, max 32), and --timeout=30 --retries=2. Without the type check flags HTTrack handles about one URL per second no matter how many connections it has.
  • Complete archive. --userdef-cmd "tar --append ..." is removed. The .tar.gz is created with a single tar call after the crawl. It leaves out the HTTrack cache (*.zip, ref), hts-in_progress.lock and unfinished *.tmp files, and keeps hts-log.txt and hts-cache/new.txt (the list of crawled URLs). --do-not-log is removed.
  • No busy loop. Waiting for HTTrack and tar polls once per second instead of spinning.
  • Locking and recovery. earkiv:process holds the earkiv_process lock (15 minute timeout, extended every minute while HTTrack or tar runs). A job still marked as running when the lock is free was abandoned and is marked for resume. A FileException in the storage step now marks the job for resume instead of leaving it in RUNNING_STORE.
  • Real resume. A resumed crawl runs HTTrack with --continue on the existing mirror. (httrack --continue on its own fails, because HTTrack cannot parse the empty footer it stored in its own cache, so the full command is repeated.)
  • Clean-up. The working directory is deleted once the archive is created. The archive is created directly in the temporary directory.
  • Exclude URLs. New setting for the HTTrack archiver, one pattern per line, passed to HTTrack as -pattern filters.
  • Archive::getFile() no longer fails when the archive has no file yet. The storage step creates the target directory if it is missing.

Testing

Tested in DDEV against a Drupal 11 site with about 2,000 published nodes and 9,500 files.

  • Depth 2 from the front page: 272 s for 272 files with the old flags, 68 s for 271 files with the new flags. No empty files, same size.
  • A crawl of a section of the site (about 700 URLs) ran through crawl, archive and storage. The job ended as Done, the archive held 702 files with no empty or *.tmp files, and /tmp was cleaned up. 400 sampled .html files were all text/html, so --check-type=0 did not mislabel anything there.
  • Killed Drush and HTTrack with SIGKILL during a crawl. The next run refused to start while the lock was held. After the lock was released the job was logged as abandoned, retries went to 1, and HTTrack continued with --continue from the URLs already fetched.
  • Drush CPU while waiting went from 99.9% to under 1%.
  • tar --append per file: 23 s for 6,000 files, compared with 32 ms for one tar -cf.

Not tested: a complete crawl of the whole site with the new code.

Notes for reviewers

  • HTTrack processes links in order. One slow URL can still stall the crawl once the prefetch window is full, whatever the number of connections. On the test site Drupal oEmbed URLs (/media/oembed?url=https://vimeo.com/...) took 30 to 140 s each because the server fetches data from Vimeo. Sites can exclude */media/oembed?* with the new setting.
  • --disable-security-limits makes HTTrack log a warning in hts-log.txt. The number of connections is still capped by the new setting.
  • The new $lock properties have no doc comments, like the existing properties in the same classes.

AI-Generated: Yes (Used Claude Code to measure the performance problems, write the code changes, run the tests in DDEV and draft this description.)

Merge request reports