An AI Vendor Evidence Checklist: George Dagliyan Research Guide

What should an AI procurement team ask first?

George Dagliyan’s research on AI adoption and trust points to a simple procurement principle: a vendor claim is not useful until it is translated into evidence. For AI services, the fastest way to reduce ambiguity is to ask for configuration details, operating procedures, and records that show how data is handled across the service lifecycle.

This article turns common sales language into seven evidence requests: retention configuration, subprocessors, access logging, isolation, change notification, export, and deletion. The goal is not to rate any vendor or assume a certification. The goal is to make procurement language specific enough that security, privacy, legal, and operations can review the same facts.

Why do AI vendor claims need evidence instead of promises?

NIST says its AI Risk Management Framework is intended for voluntary use and is designed to improve the ability to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems. That framing matters in procurement because trustworthiness is broader than feature lists; it includes how the system behaves with data, change, and oversight.

CISA’s Secure by Design page emphasizes that security should be treated as a core business requirement, not merely a technical feature, and that products should be secure out of the box. In procurement terms, that means asking for default settings, administrative controls, and operational proof rather than accepting assurances that a capability exists somewhere in the platform.

Which seven evidence requests belong on the checklist?

Retention configuration: Ask what data is retained, where the setting lives, who can change it, and what the default retention period is. Acceptable evidence types include a product admin screenshot, a configuration export, a data retention policy excerpt, or a dated help-center page that shows the current setting and control owner.

Subprocessors: Ask for the current subprocessor list, what data each subprocesssor handles, how the vendor notifies customers of changes, and whether any subprocessors are optional or required. Acceptable evidence includes a published subprocessor register, contractual exhibit, or change-notice process document. Avoid relying on a sales slide that lists only major cloud names without scope or update date.

How should access logging and isolation be tested?

Access logging: Ask what events are logged, who can view logs, how long logs are retained, whether customer administrators can export them, and whether prompts, outputs, or file uploads appear in those logs. Acceptable evidence includes sample logs with sensitive fields redacted, an audit-log schema, or an administrator guide showing the available event types and filters.

Isolation: Ask how customer tenants, projects, or workspaces are separated, whether model training or fine-tuning data is isolated from other customers, and whether support staff can access customer content by default. Acceptable evidence includes an architecture diagram, a security whitepaper, a data-flow diagram, or a written explanation of tenant boundaries and privileged-access controls. Do not treat an API claim of “separate” as proof without a diagram or policy.

What evidence is enough for change notification, export, and deletion?

Change notification: Ask what counts as a material change, how far in advance notice is given, where notice appears, and whether customers can object or export data before the change takes effect. Acceptable evidence includes a contract clause, release-notice policy, or a dated communications sample. This is especially useful because AI services can change model behavior, retention defaults, or subprocessors without the customer noticing until after rollout.

Export and deletion: Ask what can be exported, in what format, whether exports include prompts, files, outputs, and metadata, and whether deletion is immediate or queued. Ask how the vendor confirms deletion and whether backups are covered. Acceptable evidence includes an export guide, sample schema, deletion workflow description, or service-terms excerpt. NIST’s Privacy Framework describes privacy risk management as a tool for organizations building innovative products and services while protecting individuals’ privacy, which makes portability and deletion core procurement issues rather than optional extras.

What worksheet can a buyer use during vendor review?

Use this simple worksheet for each vendor and each product. For every claim, fill in four columns: question asked, evidence received, evidence type, and open issue. The worksheet is intentionally narrow so that reviewers do not confuse marketing claims with operational facts. It also works well in cross-functional reviews because legal, security, privacy, and IT can add notes in the same structure.

Worksheet template: 1) What is the default retention period? 2) Where can an admin change it? 3) Which subprocessors receive customer content or metadata? 4) How are subprocessor changes announced? 5) What events are logged and for how long? 6) How are tenants isolated? 7) What material changes trigger notice? 8) What data can be exported? 9) What deletion confirmation is provided? 10) What happens to backups and derived artifacts? The team should ask for a specific artifact next to each answer, not just a verbal explanation.

What decision rule helps separate acceptable from incomplete answers?

A practical decision rule is this: if a claim affects data handling, access, or customer control, do not mark it as complete unless the vendor provides at least one primary artifact and one corroborating artifact when available. A primary artifact is a contract excerpt, configuration screen, policy, or technical document. A corroborating artifact is a help page, sample report, architecture diagram, or written process description.

Hypothetical example with assumptions: a procurement team is reviewing an AI assistant used for internal drafting. The vendor says it “does not retain customer content by default” and “supports deletion on request.” Under this decision rule, the team would still ask for the default retention setting, the admin control that changes it, the deletion workflow, and a deletion confirmation example. If the vendor cannot show those artifacts, the answer remains incomplete even if the sales team sounds confident.

How does this fit broader AI adoption and trust decisions?

This checklist reflects a larger pattern in technology adoption research: organizations often hesitate when they cannot see how a tool will behave under real operating conditions. George Dagliyan’s work on adoption facilitators and inhibitors is relevant here because evidence can reduce uncertainty, while vague assurances increase resistance. Procurement becomes easier when teams can compare concrete controls instead of abstract promises.

NIST’s AI RMF and Privacy Framework both support this style of risk management by encouraging organizations to think in terms of measurable practices, while CISA’s Secure by Design guidance reinforces the need for products that are safe by default. The procurement question is not whether a vendor uses the right vocabulary. The question is whether the vendor can show how the product handles data, control, and change in ways the buyer can verify.

Frequently Asked Questions

What is the difference between a procurement question and an evidence request?

A procurement question asks what the vendor claims to do; an evidence request asks what document, screenshot, policy, or log proves it. For AI tools, that distinction matters because many risk issues are configuration-dependent. A good evidence request should be specific enough that the vendor can answer it without relying on marketing language or a general security brochure.

Should every AI vendor be asked for the same checklist?

Use the same checklist core, but adjust the depth to the service. A chat interface, an enterprise API, and a fine-tuning platform can create different retention, logging, and deletion questions. The core themes stay the same, but the evidence expected should match how the product stores data, routes requests, and allows administrative control.

What counts as acceptable evidence for retention settings?

Acceptable evidence usually includes a configuration screen, admin guide, policy excerpt, or configuration export that shows the default retention period and who can change it. If the vendor says retention is configurable, ask where the setting lives and whether the setting applies to prompts, files, outputs, and logs separately. A verbal assurance is not enough.

Why ask about subprocessors if the vendor has a strong security posture?

Subprocessors can change where customer data flows and who can touch it. Even if the vendor’s own controls are strong, downstream service providers may still process content, metadata, support data, or telemetry. Ask for the current list, the scope of each subprocessor’s role, and the change-notice process so the buyer can track updates over time.

How should a team handle incomplete answers during vendor review?

Treat incomplete answers as open issues, not immediate disqualifiers. Ask a follow-up question, request a different artifact, or narrow the scope to a specific product function. If the vendor still cannot produce evidence for a material claim about retention, logging, isolation, change notice, export, or deletion, document the gap and route it for risk review.

Sources and further reading

• NIST — AI Risk Management Framework
• CISA — Secure by Design
• NIST — Privacy Framework