Add test coverage

Chris Caldwellrequested to merge
issue/crawler_rate_limit-3349331:3349331-add-tests into 3.x

Closes #3349331

Merge request reports