Buying an AI Tool
How to test an AI tool before buying: the job comes first
Quick answer: run the real job with real inputs in the free trial, with the person who does the job running it. Time the job by hand first. Then rate fourteen questions: does the output need light edits or a rewrite, does it cope with your worst input, does the same input give the same answer three times, does the output land where it needs to go, and would the person use it in front of a client. This is the weight-five criterion. Nothing else matters until it passes.
5 out of five.
5 out of five.
Why the job is weighted five
The job is the only criterion that decides whether a tool belongs on the list at all. A tool can be cheap, secure, loved by the team and easy to leave, and if it does not do the job it is a subscription with no purpose. Every other criterion in this method is a way of choosing between tools that already do the job. So the job is scored first, from evidence, and it carries a weight of five.
Evidence means the trial. Not the demo. The demo is the vendor's inputs on the vendor's terms, and every product looks like it does the job in its own demo. The trial is your inputs, run by the person who will do this every week, and it is the only thing worth scoring.
Set up the trial properly
Three things before you start. First, time the job by hand once and write the number down. Question ten needs a baseline, and "it feels faster" is not one. Second, collect the inputs: the real files, the real email thread, the real spreadsheet, including the ugliest example you have. Third, give the trial to the person who does the job. If the buyer runs the trial, the buyer will score it, and the buyer does not do the job.
Then run the job the way it would be run on a Tuesday. No prompt engineering session first. No vendor on the call. If it needs an expert to get value, that is a finding for the adoption criterion, but it is also a finding here: the job did not get done by the person who does it.
The three tests that catch most of it
Fourteen questions is the full list. Three of them do most of the work.
The AI Tool Buyer's Playbook
Six gates. Four weighted criteria. Five tools. $49, one download.
Get the playbook · $49- The worst input. Give it the hardest example you have: the forty-page contract, the spreadsheet with merged cells, the email thread with six people and two languages. Did it cope, or collapse? Tools that only handle clean inputs handle a job that does not exist.
- The same input three times. Run one input three times. Three usable answers, or three different ones? A tool that is inconsistent is a tool someone has to check every time, which is most of the time you were trying to save.
- The chat window test. Where did the output land? In the document, the CRM record, the email draft, where it is needed? Or in a chat window, from which someone copies it? Every copy is a step, and steps are where adoption dies.
Then the sharpest question on the list, number fourteen: if this tool vanished in six months, what would you have lost? If the honest answer is "some time," that is a rung-one job and the bundled tool will do. If the answer is "the proposals would stop going out," you have found a tool that does the job.
Scoring it
Every question is rated one to five for each tool that survived the gates. The average of the fourteen ratings, multiplied by five, is the criterion score, out of twenty-five. The Excel scorecard does the arithmetic. The workbook has the rating circles for the kitchen table.
Rate from the trial notes, not from memory. Leave a question blank if the trial did not test it, and note that it was not tested. A blank is honest. A guess is a score that will be defended in the meeting by whoever guessed it.
Score the bundled tool too
Run the assistant you already pay for through the same fourteen questions. It will usually lose on the worst input and win on where the output lands, because it is already inside the document. That trade is the whole decision for a lot of firms, and it only shows up if the bundled tool is on the scorecard as a real option rather than the thing you are trying to replace. If it scores within a few points of the best point tool on the job, the adoption criterion will probably decide it, and the bundled tool has a head start there too.
What a low score usually means
A tool that scores under three on average here is usually a rung problem, not a tool problem. Either the job is narrower than the tool (a general assistant being asked to do something a point tool does in one step) or wider (a point tool being asked to think). Go back to the ladder before going back to the market.
Signs to walk away
- The output lands in a chat window and has to be copied everywhere.
- The same input gives three different answers, and not in a good way.
- The trial is a guided demo, not your files on your terms.
What to check yourself
- Time the job by hand once before the trial. Write the number down.
- Give the trial to the person who does the job, not the person buying it.
- Run one input three times. Three usable answers or three different ones?
The series
- How to choose an AI tool for your business without buying five
- How to test an AI tool before buying: the job comes first
- Why employees don't use AI tools, and how to score for it before buying
- AI tool data privacy checklist for a small business: the trust criterion
- AI tool pricing, per seat vs usage: how to get to one annual number
Frequently asked questions
Who should run the trial?
The person who will do the job every week. Not the owner, not the person who suggested the tool. The trial is scored by the person who ran it, and only the person who does the job can say whether the output is usable.
How long should the trial be?
Two weeks of real use, at the job's real frequency. If the job happens twice a week, that is four runs per tool, which is enough to answer every question on the list.
What if the vendor does not offer a free trial?
Ask for one. If the answer is a demo instead, treat it as a failed trial and score the job from the demo only if you supplied the inputs. A vendor that will not let you test the job on your data has answered a trust question too.
Should I test with real client data?
Only after the data gate has passed. If the tool trains on inputs or stores them somewhere you cannot name, use anonymised versions of real files. Never test with data you would not be comfortable explaining to the client.
What does "light edits" mean in question one?
Under five minutes to make the output usable. Heavy edits is more than the time it would have taken to write it. A rewrite means you used none of it. Rate five, three and one respectively.
Is speed the point?
No. Question ten checks speed, including the checking time. Question thirteen asks whether it improved the job or only sped it up. A faster version of a mediocre proposal is a mediocre proposal that arrives sooner.
How do I score a tool that does part of the job brilliantly?
Score the job as defined in the shape. If the tool does a third of it brilliantly, the job did not get done, and the score should say so. Then ask whether that third is a job on its own. It might be a separate decision.
What if two tools score the same on the job?
Good. Move to the adoption, the trust and the deal. A tie on the job means the choice is about people, terms and money, which is exactly what the other three criteria measure.
The list is not the decision. It is how you find the one or two things that are.
