How do you collect 100 real questions?
The test is only as good as the questions. Use the ones your people really ask, not the ones you wish they asked.
Spread them across who asks
Think about the roles that will use the assistant: new starters, support staff, sales, managers. Each role asks different things in a different way. Make sure every role is in the set.
How to do it: Ask a few people from each role to send you what they asked this month.
Spread them across your documents
Do not let one popular manual supply the whole set. Draw questions from every manual, policy and guide you want the assistant to use. For each question, write down which document should answer it.
How to do it: Make a short list of your documents first. Then check that each one has questions against it.
Take them from real sources, exactly as typed
Use emails, chat messages and support tickets. Do not tidy them up. Real people write in half sentences, with typos and with the wrong word for things. The assistant has to cope with that.
How to do it: Copy and paste. Resist the urge to fix the spelling.
Include some questions your documents do not cover
Add a handful on purpose. A good assistant says "I don't know" to these and points the person to a human. A poor one makes something up. You only learn which kind you have if you ask.
How to do it: Mark these as "not covered" on the sheet before the test, so you know what the right result is.
How do you mark the answers?
Give every answer one of three marks:
- Right. The answer is correct and matches your document.
- Wrong. The answer is incorrect, half right in a way that would mislead, or invented.
- "I don't know." The assistant declined to answer.
For every "I don't know", add one more note: could it have answered? If the documents did contain the answer, that reply is a miss, not a success. If the documents did not contain it, "I don't know" is exactly right.
A person does the marking, not another AI. Someone who knows your business has to read each answer. And check the source the assistant shows. Open the document it points to and see whether it really says that. An answer with no source, or a source that does not back it up, is a warning sign.
Why agree the pass mark before the test?
If you decide what "good enough" means after you see the results, the result gets argued. People who want the project will find the score fine. People who doubt it will find it too low. Writing the pass mark down first removes that argument.
Agree it in writing, with the people who will decide whether to go ahead. We do not suggest a number, because the right one depends on what a wrong answer costs you. Decide these things instead:
- How many wrong answers are acceptable. An internal help tool can tolerate more than one that talks to customers.
- Which kinds of wrong answer are deal-breakers. For example, a wrong answer about a legal rule, a price or a safety step may fail the test on its own.
- How wrong "I don't know" replies count. An assistant that refuses everything makes no wrong answers, but it is useless. Decide how many misses you will accept.
- What happens to the questions your documents do not cover. Say now whether "I don't know" is a pass on those, and whether an invented answer is a fail.
Is there a free scoring sheet?
Yes. Download the scoring sheet (CSV). It opens in Excel or Google Sheets. It has 100 numbered rows, empty and ready to fill. If you would rather start from ready-made questions, try the free company documents question pack. The columns are:
- Question (as typed): the real question, untouched.
- Asked by (role): new starter, support, sales, manager and so on.
- Document that should answer it: the manual or policy that holds the answer.
- Is it covered by the documents? yes or no, filled in before the test.
- Assistant's answer: pasted from the assistant.
- Source shown: the document it pointed to.
- Mark: right, wrong or "I don't know".
- If "I don't know": could it have answered? yes or no.
- Notes: anything the marker wants to remember.
What do you do with the result?
Compare the marks with the pass mark you wrote down. Then there are three honest choices:
- Go on to the full build. The result met the pass mark, and you saw the cases that went wrong.
- Fix and retest. The result was close. Look at the wrong answers and the wrong "I don't know" replies, change what needs changing (often the documents themselves), and run the same 100 questions again.
- Stop. The result was far off, or the documents are not ready. It is fine to stop. You spent a little, not the full price, to find out.
For how a builder should cut down made-up answers, see how to stop an AI assistant making up answers. For what to check in the company doing the work, see how to choose an AI consultancy.
Can Northstr run this test with you?
Yes. This homework is what our Hundred Questions test is built around. It costs $750 and takes 1 week. If you then go on, the full build is from $3,000, about 3 more weeks. Your fixed build price is in your Hundred Questions report. You do not have to buy it.
How we measured
- We used this kind of test on an AI tutor for an education company in a regulated industry. A person read every reply.
- On the 35 test messages about terms in the material, run before and after our changes, wrong "I don't know" replies, where the assistant refused a question it could have answered, went from 6 to 0.
- That is one assistant on one set of documents. It is not a guarantee for yours, which is why you test on your own.
Next step
Want a person to check every answer on your own documents? Hundred Questions is a one-week test: we build it, your team sends 100 real questions, and you see the results before you pay for the full build.