Loading
test: #3633124 Add functional tests for Varbase AI Safety
Issue: https://www.drupal.org/i/3633124
Adds the Varbase functional testing suite (@vardot/varbase-e2e) and a varbase-e2e-test job: a fresh Varbase 11 site, the recipe applied twice, 12 scenarios, no AI key.
- CI checks the apply stops without a default model, then uses the AI module test provider (EchoAI) and installs ImageMagick.
- 1
@ai-providerscenario(s) run in their own job only whenOPENAI_API_KEYis set.
Scenarios (13):
01-01-ai-safety-stack.feature
- Check the prompt safety and PII guardrail sets are there
- Check the guardrails cover script injection, medical advice and email addresses
- Check AI requests are logged and kept for 90 days
- Check the AI log and the observability settings open
- Check the image safety rule is on the Image media type
01-02-ai-safety-image-upload.feature
- Verify an image the vision model does not approve is refused with the safety message
- Verify a safe image is accepted (
@ai-provider)
01-03-ai-safety-log.feature
- Check the image check from the upload was written to the AI log
01-04-ai-safety-access.feature
- Check the "Content editor" user cannot open the guardrails or the AI log
- Check the "Content admin" user cannot open the guardrails or the AI log
- Check the "SEO admin" user cannot open the guardrails or the AI log
- Check the "Site admin" user cannot open the guardrails or the AI log
- Check an anonymous visitor cannot read the AI log
Green locally with gitlab-ci-local: 12 passed.
AI-Generated: Yes
Checkpoints:
- File an issue
- Addition/Change/Update/Fix
- Testing to ensure no regression
- Automated unit testing coverage
- Automated functional testing coverage
- UX/UI designer responsibilities
- Readability
- Accessibility
- Performance
- Security
- Developer Documentation
- User Guide Documentation
- Reviewed by human
- Code review by maintainers
- Full testing and approval
- Credit contributors
- Review with the product owner
- Release notes snippet
- Release
Edited by Rajab Natshah