feat: #3636844 Allow up to 128 connections and recommend a value for production servers

Thomas Alsénrequested to merge
issue/earkiv-3636844:3636844-allow-up-to into 1.0.x

Closes https://www.drupal.org/project/earkiv/issues/3636844

What it does

  • HTTrackArchiver::MAX_CONNECTIONS is raised from 32 to 128.
  • validateConfigurationForm() checks that the number of connections is a whole number from 1 to 128, so drush earkiv:create --connections gets the same check as the form. Before, 0, -1 and abc were saved from drush.
  • A stored value outside the range, for example after a configuration import, is used as the nearest allowed value with a warning in the output and the log, instead of silently.
  • The field description and the README describe the load (about 0.1 URLs per second per connection) and recommend 16–32 against a production server outside office hours, with higher values against an internal address or a copy of the site. The README has the measurements below.
  • --connection-per-second is removed, and the comment in buildCommand() explains what limits the speed.

Measurements

Full crawl of a site with about 7,400 URLs and 2.9 GB, with the flags of the module:

Connections Requests per second Time
8 1.5 83 min
32 3 38 min
64 6.5 19 min
128 13 10 min

64 and 128 connections gave identical files. 600 URLs with 64 connections took 40–41 s both with --connection-per-second=320 and without it.

Testing

  • drush earkiv:create --connections= with abc, 0, -1, 129 and 12.5 fails with "must be a whole number from 1 to 128"; 128 is accepted.
  • A stored value of 200 gives --sockets=128, and 0 gives --sockets=1.
  • phpcs (Drupal, DrupalPractice) reports no new findings compared with 1.0.x.

AI-Generated: Yes (Used Claude Code to measure the crawl, write the change and write this merge request.)

Merge request reports