Fix access check on the term

In case the term had the fields for allowing users and roles, but the current user doesn't have a role or is specifically allowed, we're not allowing the user to see.

Also: in case a user is specifically allowed, we're not returning.