Buy an operating relationship, not a demonstration
AI procurement is difficult because capability changes quickly and the buyer may be evaluating both a product and an operating relationship. Feature lists reveal little about performance on the company’s own work, the quality of evidence available to reviewers, or what happens when data, models, pricing, and service terms change.
An evidence-led approach to Coffee Cup AI begins with the workflow and continues through exit planning. The goal is to select a service the organization can understand, govern, measure, and replace if necessary. This workbook organizes questions into six lenses that produce comparable findings instead of a collection of sales impressions.
Lens one: workflow and outcome fit
Give each vendor the same representative cases and success criteria. Include normal work, exceptions, incomplete information, and a case the tool should refuse. Evaluate the final operating result, including preparation and review, rather than scoring the generated output in isolation.
Ask which steps the product changes and which remain with employees. Confirm integrations, identity controls, logging, export, and accessibility. A product that performs well in a demonstration may create excessive copying or context switching in daily use.
Evidence to collect
- Representative task test
- Total workflow effort
- User and accessibility fit
Lens two: data and security
Document what data enters the service, where it is processed and stored, how long it is retained, and whether it is used for model training. Review encryption, authentication, role-based access, audit logs, subprocessors, incident notification, deletion, and customer-controlled configuration.
Separate contractual promises from configurable settings. Verify which protections are enabled by default, which require a higher plan, and who inside the company must maintain them. Consider data residency and cross-border requirements where relevant.
Evidence to collect
- Data flow
- Access and logging
- Retention and deletion
Lens three: evidence and controls
Ask how the vendor evaluates quality, drift, safety, and bias for the relevant use. Request documentation that explains model limitations, source traceability, version changes, and customer monitoring options. General benchmark claims should not replace evidence from representative business cases.
Test abstention, escalation, and rollback. Determine whether the buyer can set thresholds, require approval, restrict actions, and inspect sources. Control should remain practical when volume increases, not depend on a single administrator remembering a manual step.
Evidence to collect
- Evaluation method
- Material change notice
- Customer control surface
Lens four: economics and service
Model price at expected volume, including premium features, integrations, storage, support, implementation, and review effort. Understand how usage is metered and which behavior could create unexpected cost. Compare annual commitments with the evidence available at the decision point.
Review service levels, support channels, incident communication, implementation responsibilities, and training. Identify the named owner on both sides. A lower subscription price can become expensive when the customer must build missing controls or troubleshoot a poorly documented integration.
Evidence to collect
- Total operating cost
- Service obligations
- Scale assumptions
Lens five: portability and exit
Confirm that the company can export prompts, configurations, logs, documents, and relevant results in usable formats. Ask how deletion is verified after termination and what assistance is available for transition. Document dependencies on proprietary agents, connectors, or data structures.
Create an exit trigger before signing. Material price change, persistent performance decline, security failure, inadequate support, or loss of strategic fit may justify transition. An exit plan improves negotiating discipline even if the relationship remains strong.
Evidence to collect
- Data export
- Deletion verification
- Transition trigger
Lens six: final decision record
Summarize evidence, unresolved questions, negotiated conditions, residual risks, and the business owner’s decision. Scorecards can support comparison, but narrative judgment should explain why a strength matters and whether a weakness can be controlled.
Set a review date tied to operating evidence. Procurement is not complete at signature. The organization should revisit performance, cost, incidents, support, model changes, and user experience before renewal or expansion.
Evidence to collect
- Decision rationale
- Conditions of approval
- Renewal review date
Questions leaders ask
What should an AI vendor pilot include?
Use representative cases, exceptions, refusal tests, total workflow effort, quality measures, security configuration, and employee feedback.
Are vendor benchmarks enough?
No. Benchmarks provide context, but the buyer needs evidence from its own tasks, data conditions, controls, and consequences.
Why evaluate exit before purchase?
Exit planning reveals lock-in, data portability, deletion, transition cost, and dependencies while the buyer still has negotiating leverage.
The operating takeaway
Strong procurement turns uncertainty into documented questions, representative tests, negotiated controls, and a reviewable decision. It selects not merely a capable model, but an operable business relationship.