AI & Automation · Digital Marketing

Questions to Ask an AI Automation Supplier Before a Pilot

Prepare an AI supplier pilot with clear workflow boundaries, data flows, representative evaluation, review responsibilities and an exit arrangement.

Published

Questions to Ask an AI Automation Supplier Before a Pilot

An AI demonstration can make a complicated business task look remarkably simple. A document arrives, the system extracts its contents, and a polished response appears within seconds. The more useful question is what happens when the document is incomplete, the answer is uncertain or the person responsible for checking it is unavailable. Those circumstances reveal whether a supplier understands the working process behind the demonstration.

A pilot should help your business make a specific decision about a specific workflow. It needs a clear boundary, representative examples and evidence that you can inspect. Before commissioning one, ask questions that connect the technology to your customers, staff and operating responsibilities. The answers should become part of the pilot brief, rather than remaining reassuring statements from a sales conversation.

Frequently asked questions

What decision will the pilot help us make?

Start with the operational problem. Perhaps staff spend too long gathering information for quotations, searching internal guidance or sorting incoming documents. Describe the current task, who performs it and where time or quality is lost. Ask the supplier which part of that task they propose to improve, and which parts will continue to require human judgement.

A proposal assistant, for example, might assemble approved service descriptions and highlight missing requirements. That is a different assignment from independently setting prices or making commitments to customers. Record the distinction explicitly. Without it, a successful experiment in drafting can gradually become an untested system for commercial decisions.

Ask what evidence would support proceeding and what evidence would justify stopping. A pilot that only promises to explore possibilities is difficult to evaluate. A useful brief names the task, the intended users, the expected improvement and the conditions under which the business will review the result.

Where will our information travel?

Ask for a practical data flow that begins with the original source and ends with the final output. Include upload tools, document storage, model providers, integration platforms, logs and any human review interface. The diagram should explain where information is copied or retained, rather than simply showing a cloud labelled with the supplier's name.

Request clear answers about the specific products and configurations proposed. Who can access submitted information? How long will each component retain it? What happens to diagnostic records and backups? Can submitted material be used to improve any provider's models, and what contractual or configuration controls apply? Answers about one product should not be assumed to cover another.

Choose pilot material deliberately. You may be able to test parts of the workflow with invented examples before introducing operational information. If real records are needed, decide which fields are necessary for the task and involve the people responsible for your information handling. Removing a name alone does not necessarily remove identifying context from a detailed document.

Which other providers does the solution depend on?

A supplier may build the workflow while relying on other businesses for models, hosting, document extraction or automation. Ask them to identify those dependencies and explain who manages changes, incidents and access problems. A single commercial contact is useful, but it does not remove the underlying supply chain.

The voluntary NIST AI Risk Management Framework organises work around governing, mapping, measuring and managing. It can help structure responsibilities and third-party risk discussions. Adapt it to the context rather than treating it as a certificate that proves a supplier's suitability. Our AI risk and procurement guide provides the related checks.

Turn that discussion into specific questions. If an external service changes its model or becomes unavailable, will the pilot stop, use a different service or return the task to staff? Who approves that change? What notice and testing will accompany it? These answers matter more than a general promise that the architecture is flexible.

How will we test quality against real work?

Ask to agree the evaluation set before judging the pilot. Include ordinary cases and examples that expose the task's difficulties: ambiguous requests, old documents, conflicting information, missing attachments and questions outside the approved scope. Keep some examples for evaluation rather than using all of them during development.

Define quality in terms of the business outcome. For a quotation assistant, check whether it identifies missing information, selects the correct approved material and avoids unsupported commitments. Attractive wording is one criterion, but it should not hide incorrect facts. A fluent paragraph that invents a delivery date can create more work than a rough draft that clearly asks for clarification.

Decide who judges the examples and how disagreements are resolved. Where experienced staff disagree about the correct response, clarify the underlying policy or accept that the task needs judgement. Recording an unresolved business question as a software defect can lead the supplier to automate an answer that your organisation has never approved.

What will reviewers see and be able to change?

Human review needs a usable interface and enough context to be meaningful. Ask whether reviewers can see the source material, identify which facts were used, correct the output and explain a rejection. A button marked approve does not establish that the person has had the information or time needed to assess the result.

Explore how uncertain or unsupported outputs are presented. The workflow might highlight missing fields, prevent certain actions or route a case to someone with appropriate expertise. Ask the supplier to demonstrate those paths with difficult examples. An explanation that staff can always intervene is incomplete if the system has already sent the message or updated a customer record.

Set the action boundary separately from the generation boundary. Producing a draft, saving it internally and sending it externally are different permissions. A cautious first pilot may generate material for review without taking the final action. Expansion should depend on evidence about the specific action being considered.

What happens when something goes wrong?

Walk through a failed journey from the user's perspective. A document upload might succeed while processing fails. A notification might disappear. A repeated request might create two records. Ask how staff recognise these states, find the affected case and recover without guessing whether an action has already happened.

Agree a named route for reporting problems, an owner for operational decisions and a fallback process that the team can actually use. Where the workflow replaces an existing manual step, keep the necessary access and instructions available during the pilot. The fallback should be rehearsed with an example rather than described only in the proposal.

Also ask what information will be recorded for investigation. Useful records connect the input, system version, output and action taken, while respecting your information handling requirements. Avoid collecting everything indefinitely simply because logging is easy. The evidence should answer a defined operational question.

Can we understand costs and leave the pilot?

Separate setup work, ongoing support and usage charges. Ask which events create usage, including retries, document processing, storage and evaluation. Request examples based on plausible business activity, with assumptions stated. A headline cost per task can be misleading if a task triggers several services or needs repeated attempts.

Discuss what you will receive at the end: workflow documentation, evaluation results, relevant configurations, exports and an explanation of remaining limitations. Ask which accounts your business will control and how access will be removed when the work ends. Establish the agreed arrangements before relying on the workflow for daily operations.

Portability deserves a concrete demonstration. Can you export the approved knowledge, retrieve records in a usable format and understand the integrations another developer would need to maintain? A promise of easy migration is less useful than a sample export and a clear account inventory.

Who will decide what happens after the pilot?

Book the review before starting. Give someone responsibility for combining operational feedback, evaluation results, costs and unresolved issues. The decision might be to expand, revise the scope, run another limited test or stop. Each is a valid result when supported by evidence.

Keep the review focused on the original task and the consequences of expanding it. Success with internal drafting does not automatically support autonomous customer communication. Wider use needs its own boundaries and checks.

If you need to connect an AI idea to your existing tools and customer journey, Giraffe Digital's digital strategy service can help frame the business requirements. A well-prepared supplier conversation turns a promising demonstration into a testable proposal, with clear responsibilities and a decision your team can explain.

AI & AutomationDigital Marketing