What documents would we test?
Nothing outside the documents you share would be used. For an IT services company, that usually means four kinds of document.
- Runbooks. These are the step-by-step fixes and routines: resetting access, restoring a backup, rebuilding a profile, handling an alert.
- Client documentation. Network notes, site details, contacts, escalation paths and vendor accounts, kept per client. This is where most of the risk sits, because every client is set up a little differently.
- SOPs. Your own standard procedures: onboarding a new starter, offboarding, change approval, how to hand a ticket up a line.
- Past ticket resolutions and knowledge-base articles. They show how a problem was actually fixed, often in plain words.
One rule before anything else: credentials stay out. Passwords, keys and recovery codes should stay in your password manager and out of the assistant. The documents can say where a credential lives. They should never contain it.
What would people ask it?
First line wants a quick fix. Engineers want detail. Team leads want the rule. Here are 10 sample questions across those groups, each with the document that should answer it. Your real tickets will read differently, which is why the test uses them.
| Who asks | Sample question | Where the answer should come from |
|---|---|---|
| Service desk (first line) | How do I reset MFA for a user at the dental practice client? | That client's identity notes, or the shared MFA runbook if the client has no override |
| Service desk (first line) | Who do I call at Client A after hours if the phones are down? | Client A's contacts and escalation page |
| Service desk (first line) | A user says a shared mailbox has disappeared. What do I check before I escalate? | Shared mailbox runbook and past ticket resolutions |
| Engineer (second or third line) | Which backup check runs on Client B's file server, and where do I see the result? | Client B's backup documentation |
| Engineer (second or third line) | What is the escalation path for a down firewall at the law firm client? | That client's escalation page and the firewall vendor details |
| Engineer (second or third line) | We saw this VPN error before. How was it fixed last time? | Past ticket resolutions and knowledge-base articles |
| Engineer (second or third line) | Which vendor do I raise a line fault with for Client C's main office? | Client C's site details and vendor accounts page (no logins, only who to contact) |
| Project / onboarding engineer | What are the steps to onboard a new starter at Client A? | The shared onboarding SOP, plus Client A's overrides |
| Project / onboarding engineer | What do we need to collect from a new client before we take over their network? | Client takeover SOP |
| Team lead | Which approval does a change to a client's firewall rules need, and who signs it off? | Change approval SOP, plus any client-specific rule |
Where could it go wrong, and how would we test that?
There are two failure modes that matter most for an IT services company. Both come from how you hold documents for many clients. We would test for both on your material.
One client's documents answer for another client
Many of your clients run similar setups, so their pages look alike. If the assistant mixes them up, an engineer could reset the wrong thing or call the wrong person. The fix is separation. Answers must come only from the client named in the question or the ticket. If the client is unclear, the assistant must ask which one. Shared SOPs apply to everyone, but a client-specific override beats the shared version for that client only.
How we would test it: Ask pairs of questions where two clients do the same task differently, and check each gets its own answer. Then ask about a client whose documents do not cover the topic. The right result is "I don't know" for that client, never another client's answer borrowed in its place. A cross-client answer is an automatic fail.
It trusts a page that has gone stale
Client documentation ages quickly. A vendor changes, a new firewall goes in, a contact leaves. An old page can look just as confident as a fresh one. The assistant can only be as current as your documents, so it must show the source and its last-updated date with every answer. That lets the engineer judge it. Pages with no owner or no date should be flagged as such.
How we would test it: Include questions about things you know changed recently. Check that the reply cites the updated page and not the old one. Include a few pages with no owner or date and check that they are flagged.
Our general approach is in how to stop an AI assistant making up answers.
How would the test be marked?
An engineer who knows your clients reads every reply and marks it right, wrong or "I don't know". The person also opens the source the assistant shows and checks it says what the assistant claims, and that the date shown is the real one.
Put the pass mark in writing first. For an IT services company, it is worth deciding now that any answer taken from the wrong client is a deal-breaker. The full marking method is in how to test an AI assistant before you buy. There is also a free company documents question pack with 100 starter questions. We also have other industry guides, such as one for franchise operations manuals.
Doesn't our documentation tool already do this?
If your documentation tool already has AI search, test it on the same 100 questions first.
If it passes your pass mark, you may not need us. Hundred Questions is for when you want to compare, or when it falls short. Use the same questions and the same marking for both, so the comparison is fair.
What would you send us for a Hundred Questions test?
Your documentation is full of client detail, and that is sensible to protect. You do not have to send everything. The test can start with a small set, under an NDA if you like.
Send us your runbooks and SOPs, documentation for two or three clients with credentials stripped, and 100 real questions from tickets and chat.
- Runbooks and SOPs. Pick the ones people reach for most, plus any that have a client-specific override.
- Documentation for two or three clients. Include one client whose setup overlaps a lot with another. That is the best way to test separation. Strip out passwords, keys and recovery codes first.
- 100 real questions, copied straight from tickets and chat. Do not tidy them up. Spread them across first line, engineers and team leads.
Mark which questions the documents do not cover. There, "I don't know" is the pass. Tagging them up front keeps the scoring honest.
Your documents stay in accounts you own. Hundred Questions costs $750 and takes 1 week. If you then go on, the full build is from $3,000, about 3 more weeks. Your fixed build price is in your Hundred Questions report. It is fine to stop after the test.
Has Northstr built one for an IT services company?
We haven't built one for IT services companies yet. Our proof so far is an AI assistant for an education company in a regulated industry.
- Across 150 test messages, each read by a person, we saw no made-up answers.
- Wrong "I don't know" replies went from 6 to 0 on 35 test messages about terms in the material.
- The AI fees were $2.52 for 684 AI answers, about $0.004 each.
The details are in the AI tutor case study. One assistant on one set of documents is not a guarantee for yours. That is why we would test on your own runbooks first.
Next step
Want to see how an assistant behaves on your own runbooks and client documentation? Hundred Questions is a one-week test: we build it on a sample, your team sends 100 real questions, and a person checks every answer.