Process "paths to ignore" before bot rate limiting

Vojislav Jovanovicrequested to merge
issue/crawler_rate_limit-3528632:3528632-ignore-paths-before-bot-limiting into 3.x

Closes #3528632

Merge request reports