What documents would we test?
It would answer only from the documents you hand over. For a service team, that usually means three core kinds of document, and up to three more.
- Service manuals. They hold the repair and maintenance procedures, one manual per model line or family. Most technician questions are answered here.
- Parts lists and parts catalogs. They tie each part to a position on an exploded diagram. People ask which part to order, and whether it fits.
- Troubleshooting guides. They link a symptom or an error code to the checks and fixes, in order.
- Service bulletins (optional). They change a procedure or a part after the manual was printed. They matter a lot, and they are easy to miss.
- Wiring diagrams (optional). Mostly pictures. We would be honest about how much of a drawing an assistant can read.
- Warranty procedures (optional). They say what is covered and what to record before a claim.
Much of this material is not plain text. Answers sit in tables, in exploded diagrams and in old scanned PDFs where the page is really a photo. That is what makes service documents harder than most, and it drives the two failure modes below.
What would people ask it?
A technician at a customer site and an agent on the phone need different answers. Here are 10 sample questions, grouped by who asks, with the page that should answer each. Real ones from your tickets will be shorter and rougher. Use those in the test, not these.
| Who asks | Sample question | Where the answer should come from |
|---|---|---|
| Field technician | The control panel shows an error code on the older model line. What does it mean and what do I check first? | Troubleshooting guide, error code table for that model |
| Field technician | Which part replaces a worn door seal on this unit? | Parts list, with the exploded diagram to confirm the position |
| Field technician | What are the lockout steps before I open the service panel? | Service manual, safety section (quoted exactly) |
| Field technician | How do I reset the controller after a fault, and do the steps change on later units? | Service manual, reset procedure, plus any service bulletin |
| Bench technician / service coordinator | Is this pump the same part on both model lines, or only a look-alike? | Parts list for each line, and any superseded-part note |
| Bench technician / service coordinator | What tightening values apply to the cover bolts on this assembly? | Specification table in the service manual |
| Bench technician / service coordinator | Does the warranty cover this failure, and what do I record before ordering the part? | Warranty procedure, plus the parts list |
| Customer support | A customer says the unit won't start. What can I ask them to check before we send someone? | Troubleshooting guide, first-checks section |
| Customer support | Which model and age of unit does this fault apply to, so I know whether it is a known issue? | Service bulletins and the model identification page |
| Customer support | The customer wants to know if they can clear a fault themselves. What do I tell them? | Owner-safe steps in the manual; anything else: send a technician |
Where could it go wrong, and how would we test that?
Two failure modes matter most for a service team. Both come from how equipment documents are built. We would test for both on your manuals. A third risk, safety steps, is big enough to get its own section below.
It gives the procedure for the wrong model or serial range
Equipment families often have models that look alike. Their manuals read alike too. Worse, a procedure can change partway through a serial range, so two units of the same model need different steps. If the assistant mixes them up, a technician follows steps for a machine they are not standing in front of. When the person has not said which model or serial range, the assistant must ask. When it answers, it must cite the manual for that model.
How we would test it: Include questions that name only a product family. Include questions where two models differ. Check that the assistant asks which model or serial range, or cites the right manual for the one given. A confident answer for a guessed model is a fail.
The answer is locked in a table or a scanned page
Parts tables, torque and specification tables, and fault-code tables hold exact values in cells. Reading the wrong row or column gives a plausible but wrong answer. Older manuals may be scanned images with faint print or skewed pages, and a scan can be read badly or not at all. When the assistant cannot read the page, it must say "I don't know". It must not fill the gap with a guess.
How we would test it: Include questions whose answers live only in a table or on a scanned page. Open the page and check the answer matches the cell. Include a few that sit on the worst scans, where the right result is "I don't know". We would also tell you which documents need a clean copy.
The wider method is in how to stop an AI assistant making up answers.
What is the rule for safety steps?
Some answers carry a risk of injury: lockout, electrical isolation, pressure release, protective equipment. For these, the rule is stricter than for anything else.
Safety steps must be quoted from the manual, never paraphrased. The assistant shows the exact wording and the source page. If it cannot find the safety step, it says so and points to the manual or a supervisor.
A summary can drop a step or soften a warning. So the assistant must not summarize. It either quotes the step as written or says it cannot find it.
How we would test it: Ask for the lockout, isolation and pressure-release steps on several units. Compare the quoted text with the source word for word. Any changed wording, missing step or invented step is a fail. Also ask for a safety step that is not in the documents. The right result is "I can't find it, check the manual or ask a supervisor".
How would the test be marked?
Someone who services your equipment reads every reply and gives it one mark: right, wrong or "I don't know". The person also opens the page the assistant shows and checks that it says what the assistant claims, for the right model.
Write the pass mark down before the test runs. For a service team, it is worth deciding now that a wrong or reworded safety step is a deal-breaker. The step-by-step version is in how to test an AI assistant before you buy. There is also a free company documents question pack with 100 starter questions.
What would you send us for a Hundred Questions test?
You do not have to send every manual. The test can start with one or two model lines.
Send us a few service manuals for one or two model lines, the matching parts list and troubleshooting guide, any service bulletins, and 100 real questions from technicians and support.
- A few service manuals. Include at least one older scanned manual and one with heavy tables. They are the hard cases, so they tell you the most.
- The matching parts list and troubleshooting guide. Use the same model lines as the manuals.
- Any service bulletins that change those manuals or parts.
- 100 real questions, word for word. Pull them from tickets, call notes and messages from field technicians, bench technicians and customer support. Do not tidy them up.
Mark which questions the documents you sent do not cover. The correct reply to those is "I don't know", and flagging them first means nobody argues about the score later.
Your documents stay in accounts you own. Hundred Questions costs $750 and takes 1 week. If you then go on, the full build is from $3,000, about 3 more weeks. Your fixed build price is in your Hundred Questions report. It is fine to stop after the test.
Has Northstr built one for equipment service teams?
We haven't built one for equipment service teams yet. Our proof so far is an AI assistant for an education company in a regulated industry.
- Across 150 test messages, each read by a person, we saw no made-up answers.
- Wrong "I don't know" replies went from 6 to 0 on 35 test messages about terms in the material.
- The AI fees were $2.52 for 684 AI answers, about $0.004 each.
The details are in the AI tutor case study. One assistant on one set of documents is not a guarantee for yours, and tables and scans are a new test for us. That is why we would test on your own manuals first.
Next step
Want to see how an assistant behaves on your own service manuals? Hundred Questions is a one-week test: we build it on a sample, your team sends 100 real questions, and a person checks every answer.