What should a 20–300-person company check before hiring an AI consultancy?

What are the seven checks?

1. 1

   A fixed-price first step

   Before any big commitment, there should be a small first step with a fixed price and a fixed time. You learn whether the idea works on your material before you spend real money. If the first thing on offer is a long project with a monthly retainer, you are paying to find out what they could have shown you quickly.

   Ask them: "What is the smallest piece of work you can do for a fixed price and a fixed end date, and what will I have in my hands at the end?"

2. 2

   Tested on your own questions, checked by a person

   A demo on their questions proves nothing about yours. Ask for a test on real questions from your team or customers, typed the way people really type. Then ask who checks the answers. If another AI is the only judge, mistakes slip through. A person should read the replies.

   Here is what that looks like. On an AI tutor we built for an education company in a regulated industry, we ran the same fixed test messages before and after a round of changes, and a person read every reply. On the 35 test messages about terms in the material, wrong "I don't know" replies, where the assistant refused something it could have answered, went from 6 to 0.

   How to check it: Give them 100 real questions you wrote yourself, untidied. Ask to see every reply, with a note on who read it and how they scored it. Our guide on how to test an AI assistant before you buy shows how to run this yourself.

3. 3

   It says "I don't know" and shows sources

   An assistant that must always answer will sometimes make things up. A good one answers from your documents, shows which document each answer came from, and says "I don't know" when the answer is not there. A good test counts two mistakes: made-up answers, and wrong "I don't know" replies. Counting only one rewards an assistant that either guesses or refuses everything.

   How to check it: Ask it questions your documents do not cover. It should decline them. Then open the source it shows for a few answers. Does the source really say that?

4. 4

   You own the accounts

   The AI provider accounts, the hosting, and the documents should be set up in your name and billed to you. At the end they are handed over, with the keys. If the consultancy holds everything under its own accounts, you cannot leave without starting again.

   Ask them: "Whose name is on the AI provider account and the hosting account? Will the bills come to me directly? What do I get at handover?"

5. 5

   Cost per answer in writing

   A monthly estimate hides the number that matters. Each answer costs a small amount in AI fees, and the total depends on how many answers your people ask for. Get the cost per answer, plus what is included in the running cost, in writing. Our guide to what an AI assistant costs explains how to read it.

   Ask them: "What does one answer cost in AI fees, measured on a real test, and what happens to the bill as usage grows?"

6. 6

   They name which AI provider sees your text

   When someone asks the assistant a question, your document text goes to an AI provider to produce the answer. You have a right to know which one, and where your documents are stored. A vague "it's secure" is not an answer.

   Ask them: "Which AI provider receives my text? Where are my documents stored? Is my text used to train anything?" Get the reply in writing.

7. 7

   You can stop after the first step

   After the first step you should be free to walk away, and still keep the report. No lock-in, no penalty, no "you need the next phase to use what we found." If the work is good, you will want to continue. If it is not, you should not be stuck.

   Ask them: "If I stop after the first step, what do I keep, and is there any cost or notice period?"

What are the red flags?

• There is no fixed first step, only a long project or a retainer.
• They will not put cost per answer in writing.
• They cannot say where your text goes, or which AI provider sees it.
• They ask for access to your systems before they have scoped the work.
• They only show demos on their own questions.
• The accounts are in their name, not yours.

How we measured

The one example above comes from our own work. Here is where it came from and what it does not tell you.

• The 6 to 0 counts the 35 test messages that ask about terms in the material. We ran the same messages before and after the changes.
• Each reply was read by a person. We did not leave the scoring to another AI.
• It was one assistant on one set of documents, for an education company in a regulated industry.
• It is not a guarantee for your documents. That is why the first step should be a test on your own.

What is the short version?

1. Fixed-price first step, small and quick.
2. Your questions, with a person checking the answers.
3. "I don't know" and sources, built in.
4. You own the accounts, and they are handed over.
5. Cost per answer, in writing.
6. Provider named: you know who sees your text.
7. Free to stop after the first step, report kept.