How to Compare AI Vendors Properly Instead of Picking the Loudest One

A vendor demo is a rehearsed performance. The data is clean. The questions are expected. The tool works because it has been set up to work.

Your buying team sees ten minutes of polish and treats it as proof. The vendor with the best slides and the most confident salesperson gets the contract. Their product may be good. You have not checked.

techUK reported in May 2026 that buyers often lack the leverage or the technical knowledge to demand better disclosure from vendors, and that AI washing is widespread. AI washing means a supplier labels ordinary software as AI to make it sell. You cannot spot it in a demo. You spot it in a structured comparison.

Write down what you need the tool to do. Use one sentence per task. For example: reduce the time to answer a standard supplier query from 20 minutes to five.

Then decide how much each criterion matters. Give every criterion a weighting out of 100. Cost might carry 15. Security and data handling might carry 25. Fit with your existing systems might carry 20. Evidence of results might carry 25. Exit terms might carry 15. Your numbers will differ. The point is that you agree them first.

Do this before vendors contact you. Once a salesperson has shaped your thinking, you will weight the criteria that suit their product. I have watched this happen across over 30 years of running technology companies. It is human and it is predictable. A written scorecard stops it.

Use one scorecard for every vendor. Give each criterion a score from one to five. Have at least two people score independently, then compare. Where scores differ by two points or more, discuss why.

Your scorecard should cover these areas at a minimum.

Vendors prepare for price and feature questions. They are less ready for liability and drift.

Ask who pays when the system gives a wrong output that causes harm. techUK found that clear frameworks for sharing responsibility across AI supply chains are largely absent. That means your contract is your protection. If the vendor will not accept any liability, record that on the scorecard.

Ask what happens when the underlying model changes. AI products change without notice. A tool that passed your test in March can behave differently in June. Ask how the vendor tests for drift and how they tell you about changes.

Ask whether AI features can be switched on inside products you already own. techUK noted that AI can appear in products under contract with no terms governing it. Check your existing suppliers as well as your new ones.

Ask for a model card or equivalent documentation. techUK reported that model cards are rarely requested, partly because many buyers cannot interpret them. If you cannot read it, pay an adviser to read it for you. If the vendor has nothing to show, that is your answer.

Shortlist two or three vendors. Run a paid trial of two to four weeks with each. Use your own data and your own staff, not the vendor's sample files.

Write the pass mark before the trial starts. For example: at least 90 percent of outputs accepted without edits by the people who use them. If the tool misses the mark, you have your answer. Nobody argues with a number you wrote down in advance.

A paid trial costs money. A failed full contract costs far more. Budget for the trial inside the project cost, not as an afterthought.

Pick one AI purchase you are considering this quarter. Open a blank document and write down the five to seven criteria that matter. Give each a weighting out of 100.

Send the scorecard to the person who will sign the contract and ask them to agree it in writing. Do this before you book any more demos.

Add one rule to your shortlist process: no vendor goes to contract without a documented answer on liability, data ownership and exit. If a vendor refuses to answer, they leave the shortlist.

If you want a ready-made structure, use the AI vendor comparison resource linked from this article.