feat: #3636586 Improve feedback during the archiving run

Thomas Alsénrequested to merge
issue/earkiv-3636586:3636586-improve-feedback-during into 1.0.x

Closes https://www.drupal.org/project/earkiv/issues/3636586

What this changes

  • New earkiv.progress service (ArchiveProgress). Stores the progress of each job in the State API under earkiv.progress.<id>: phase (crawling, checking broken links, compressing, waiting, storing, done, failed), crawled URLs, start URLs, broken links, archive size, start, last update, finish and an error message. It also formats a one-line summary and detects stalled jobs (no update for 5 minutes). No schema change.
  • During earkiv:process the progress is updated every 5 seconds and at every phase change, and the Drush line shows it, for example Crawling: 693 URLs, 5 min 55 sec, 2.0 URLs/s. While compressing it shows the growing archive size. A resumed job keeps its start time, so the running time covers all attempts.
  • New drush earkiv:status command. Lists jobs that are new, running or waiting to be resumed (--all for finished jobs too) with status, progress, start and last update. --fields=id,status,urls,broken,size,updated adds the raw numbers. Warns when a running job is stalled.
  • Archive list. The progress is shown below the status of each job, and finished jobs show a summary. The list is not cached and reloads every 30 seconds while a job is running.
  • Log. A summary is logged when an archive is done (E-Arkiv archive 10 created: 702 URLs, 13 broken links, 10.56 MB, 6 min 5 sec.) or has failed. Abandoned jobs are marked as failed in the progress before they are resumed.
  • The progress is removed in Archive::postDelete(), so it also goes when an archive is deleted with drush entity:delete.
  • README: documents the progress and earkiv:status, and ticks the to-do item.

Testing

In DDEV, a job of about 700 URLs (6 minutes):

  • Drush showed the progress line every 5 seconds through crawl, broken link check and compression, and the summary at the end.
  • drush earkiv:status during the crawl showed Crawling: 9 URLs, 10 sec, and the URL count with --fields.
  • The archive list during the crawl showed Crawling: 41 URLs, 27 sec under the status and had the refresh meta tag. After the job it showed 702 URLs, 13 broken links, 10.56 MB, 6 min 5 sec and no refresh tag.
  • The summary was written to the Drupal log.
  • drush entity:delete earkiv_archive 10 removed the progress record.

Notes for reviewers

  • The issue proposed storing progress every 10 seconds; it is every 5 seconds, together with the Drush line. That is one small State write per 5 seconds while a job runs.
  • The crawled URL count comes from hts-cache/new.txt, where HTTrack adds a URL when the file is complete, so it lags a little behind the URLs in progress.

AI-Generated: Yes (Used Claude Code to write the code changes, run the tests in DDEV and draft this description.)

Merge request reports