Issue #3092690 by Dave Reid: Added setting to exclude items that have individually overridden the robots meta tag to noindex.