Add configurable request path allowlist

Vojislav Jovanovicrequested to merge
issue/crawler_rate_limit-3442260:3442260-configure-paths-to-ignore into 3.x

Closes #3442260

Merge request reports