The SMB Guide to Vetting AI Tools: 6 Questions to Ask Before You Automate
TL;DR
Ask about data handling, model provenance, vendor security posture, fallback processes, explainability, and costs.
The 6 questions
1. How is my data stored and encrypted?
Expect AES-256 at rest, TLS in transit. Ask for SOC 2 Type II certification and data residency options.
2. What model is used and can I restrict outputs?
Prefer vendors that allow instruction tuning or private models. Understand if your data trains their models.
3. Can we export our data?
Portability for switching vendors is critical. Ask about data export formats and frequency.
4. What is the vendor's incident response plan?
Review SLA, notification procedures, and breach history. Ask for their security whitepaper.
5. What fallback/human-in-the-loop controls exist?
For critical decisions, ensure there's a human review step. Understand confidence thresholds.
6. How will the vendor measure accuracy & bias?
Ask for ongoing monitoring reports, bias audits, and model performance metrics.
Checklist for procurement
- Add a security appendix in your contract
- Pilot with masked/anonymized data
- Require SOC2 or ISO certification where applicable
- Review vendor's sub-processors and third-party dependencies
- Establish clear data retention and deletion policies
Need help running vendor checks?
Download our vendor checklist or schedule a security briefing with our team.