feat: #3636938 Add a manifest, checksums and a start page to the archive

Thomas Alsénrequested to merge
issue/earkiv-3636938:3636938-add-a-manifest into 1.0.x

Closes https://www.drupal.org/project/earkiv/issues/3636938

One commit per part.

Commits

  1. Add a manifest to the archive. earkiv-manifest.json in the root: job ID, archived URL, when the crawl started and finished, the versions of E-Arkiv, Drupal and HTTrack, all archiver settings, the counts of URLs, files, bytes, broken links, unlinked pages and published pages with errors, the path of the start page of the site and the report files. The files left out of the archive are now one list (ARCHIVE_EXCLUDE), used by tar and by archiveFiles().
  2. Add checksums of the files in the archive. earkiv-checksums.sha256 with the SHA-256 checksum of every file in the format of sha256sum, written last so that it covers every other file. The lock is extended while the files are hashed.
  3. Store the checksum of the archive file. New base field checksum on the archive (installed by earkiv_post_update_archive_checksum_field()), set after compression, shown in the archive list, and written next to the stored archive as earkiv_….tar.gz.sha256, which is removed with the archive. Cleared when a job starts over.
  4. Replace HTTrack's index with a start page for the archive. HTTrack's index.html links to every start URL, which can be thousands. It is replaced with a page that says which site was archived and when, links to the start page of the site and to the manifest and the reports, with translatable texts. backblue.gif and fade.gif are removed.

Testing

Full archive of a Drupal 11 site: 7,876 URLs, 110 broken links, 2.39 GB, 13 min 35 s with 128 connections (about a minute more than before, mostly for the checksums).

  • The manifest had all fields, e.g. 7,882 files, 3.25 GB, start page …/index/index.html, HTTrack 3.49-6.
  • shasum -a 256 -c earkiv-checksums.sha256 on the extracted archive: all 7,883 files OK, 15 s.
  • shasum -a 256 -c earkiv_….tar.gz.sha256 next to the stored archive: OK; the same checksum is on the entity and in the archive list. Deleting the job removed both files.
  • The start page links to the site start page and the seven files; HTTrack's images are gone.
  • The exclude matcher, tested on a small mirror, leaves out the same files as tar.
  • phpcs (Drupal, DrupalPractice) reports no new findings compared with 1.0.x.

AI-Generated: Yes (Used Claude Code to write the feature, test it on a full archive and write this merge request.)

Merge request reports