fix: #3636912 Validate input and make failed jobs recoverable

Thomas Alsénrequested to merge
issue/earkiv-3636912:3636912-validate-input-and into 1.0.x

Closes https://www.drupal.org/project/earkiv/issues/3636912

One commit per item, so they can be reviewed one at a time.

Commits

  1. Require an http or https URL for the site. file:// and ftp:// were accepted.
  2. Require * or / in exclude patterns. HTTrack reads a filter without them as an option; Ox made the job fail with exit 255. The message suggests *Ox*.
  3. Resume jobs that fail before their step has started. setJobStatusErrorResume() now maps New and Resume archive to Resume archive, and Waiting and Resume storage to Resume storage, instead of Error.
  4. Give storing its own resume attempts. The count is reset when the archive has been created.
  5. Add drush earkiv:retry and a Retry operation for failed jobs. Archive::retry() stores an archive that waits in the temporary directory again, and otherwise removes the files and starts over with a new crawl. The Retry operation (confirmation form, entity update access) is shown for jobs with the status Error. Documented in the README.
  6. Do not let the last save undo a cancellation. The status is checked before the error pages are handled and right before the job is saved as waiting.
  7. Extend the lock while the archive is copied to storage. The archive is renamed when possible; across file systems it is copied in 8 MB parts to a .part file while the lock is extended every 60 s, then checked by size and renamed. A free file name is chosen like FileSystem::move() did.
  8. Start over when the waiting archive is missing. The storage step checks for the archive and, if it is gone (for example after a restart), lets the job start over with a new crawl instead of failing.

Testing

  • drush earkiv:create rejected --url=file:///etc, --url=ftp://example.com and --exclude=Ox, and accepted --exclude=*?sort=*.
  • A job whose start page returned 404 went through its resume attempts before it ended as Error.
  • drush earkiv:retry set a failed job to New, and a failed job with an archive in the temporary directory to Waiting; a job that had not failed was refused. The Retry link and the confirmation page are shown for the failed job.
  • A 25 MB archive was stored from the temporary directory to the private file system on another file system (copied in parts, no .part file left).
  • A waiting job whose archive was missing started over as New.
  • phpcs (Drupal, DrupalPractice) reports no new findings compared with 1.0.x. Items 4 and 6 were checked by reading the code.

AI-Generated: Yes (Used Claude Code to write the fixes, test them and write this merge request.)

Merge request reports