How many vendors should we shortlist before doing this?

Three is usually the right number. Two is not enough to expose the trade-offs; five takes weeks of evaluation work and produces diminishing returns. Use a desk-based filter to get from a long list to three: regulatory fit, basic feature coverage, vendor stability, and reference customers in your sector. Then put those three through the structured comparison. If two of the three fall away early on accuracy or cost, that is a clean signal, not a problem.

Can the vendor build the test dataset for us?

No. The whole point is that the dataset reflects your real inputs, including the awkward edge cases the vendor would never volunteer to be tested on. Build it from your own archive, with someone from operations choosing the cases. The vendor can be told what a correct output looks like for each case, but they should not be choosing the inputs or judging the results. If a vendor refuses to be tested on inputs they did not curate, treat that as a finding in itself.

How do we cost integration when the vendor will not commit?

Push for a written estimate based on a defined integration scope — your specific systems, the data flows, the authentication model. If they will only quote 'time and materials', use a sensible upper bound for the year-one model and flag it as a risk. Reference customers are the second source: ask two or three buyers what they actually paid versus what was quoted at the start. The gap is usually informative, and any vendor that cannot facilitate those reference calls has told you something.

What weighting should a UK council use versus an SME?

A council should weight compliance, audit trail, data residency and switching cost most heavily, because the procurement has to defend itself against challenge and the contract will run for years. An SME can usually weight speed-to-value, integration ease and total cost more heavily, because adoption risk is the bigger threat. Whatever you choose, write the weights down before you score the vendors. Setting weights after seeing the scores is how procurement decisions become hard to defend.

Do we need independent help to run this comparison?

Not always, but it helps in two situations: when the buying team has not run an AI procurement before and is being lobbied hard by vendors with skilled sales engineers, and when the decision needs to be defensible to a board, auditor or scrutiny committee. An independent reviewer enforces the methodology, asks the awkward questions vendors would prefer to avoid, and produces the scored comparison and documented rationale that makes the eventual decision auditable.

Executive Resources · for UK SME leaders

How to Compare AI Vendors Properly

Most AI vendor comparisons collapse into a battle of claimed accuracy percentages on demo environments that were configured to win. The right approach is head-to-head testing on your own data, a full year-one and year-two cost model, an operational scorecard covering integration, training and audit trail, and a structured risk assessment. Score the criteria by what matters most to your organisation, document the methodology, and the vendor that fits your problem will beat the vendor with the loudest spec sheet.

Why headline accuracy is almost meaningless

Vendor A says 95% accuracy. Vendor B says 97%. Vendor B wins. This kind of comparison leads organisations to buy the wrong tool on the strength of benchmarks that were designed to be won.

Those accuracy figures come from curated test sets the vendor controls. Your data is messier — unusual formats, multiple currencies, handwritten items, non-standard layouts, the legacy of a CRM migration nobody finished. The only number that matters is how each vendor performs against your inputs, judged against your definition of correct.

The five steps below replace the spec-sheet shootout with a structured evaluation that produces a defensible decision.

Step 1: Define the evaluation scenario using your data

Do not use vendor demos. Vendor demos are configured to show the tool at its best. Build a test dataset from your own archive — real inputs that reflect the variety and messiness of your actual workflows.

For document processing, gather fifty real examples mixing straightforward cases and edge cases: unusual formats, multiple currencies, handwritten items, non-standard layouts. Define precisely what a correct output looks like for each case. Set a minimum accuracy threshold before you begin — for example, correct extraction of four specific fields in 95% of cases. The threshold is the bar each vendor has to clear; without it, the comparison drifts into impressionistic preference.

Step 2: Run the same test on each vendor

Process your test dataset through each shortlisted vendor under identical conditions. Resist the temptation to let one vendor 'tune' the system before the run; you are measuring out-of-the-box performance against your data.

Measure four things for every vendor and record them in a consistent format so comparisons stay objective rather than impressionistic.

Accuracy: percentage of test cases with correct output against your definition.
Speed: processing time per item.
Failure rate: percentage of cases where the tool produces no usable output.
Error type: are failures random edge cases or systematic patterns that will affect your core workload?

Step 3: Total cost of ownership — not just software cost

Build a full year-one cost model for each vendor covering software licence, implementation services, integration work, staff training and annual support. Then build year-two-onwards costs, which are often significantly lower because implementation is a one-off.

Calculate the net benefit: annual labour savings plus efficiency gains minus total annual cost. The vendor with the highest accuracy on your test is not always the vendor with the best ROI. Integration complexity, training burden and support overhead can reverse a headline accuracy advantage entirely. A 70-person manufacturer choosing on accuracy alone often picks the wrong tool because the year-two cost picture was never built.

Step 4: Operational factors

Beyond performance and cost, the operational comparison covers the factors that determine how easily a tool embeds into your organisation. These are the items most often missed in spec-sheet comparisons and most often blamed when adoption stalls six months in.

Factor	What to measure	Why it matters
Integration method	API, webhook or manual	Manual integration kills adoption
Setup time	Weeks to go-live	Delayed value realisation
Training burden	Hours per staff member	Hidden cost and adoption risk
Error handling	Flags low confidence vs silent failure	Silent failures damage trust
Audit trail	Included vs paid add-on	Compliance and governance requirement
Support response	Hours for critical issues	Downtime cost in production

Step 5: Risk assessment

Score each vendor on four risk dimensions: vendor stability (financial health, market position, exit risk), data security (certifications, audit rights, breach track record), switching cost (data portability, contract exit terms) and integration risk (how deeply embedded the tool will become and how hard it is to remove later).

This is the dimension where the lowest-priced vendor often loses, and where a council buyer rightly weights compliance and audit rights more heavily than a high-growth SME would. The point of doing it explicitly is to avoid finding out which risks mattered after the contract is signed.

Scoring and decision

Weight the criteria by what matters most to your organisation. A council procurement will weight compliance and audit trail highest. A high-growth SME might weight integration ease and speed-to-value. Apply the weights consistently to produce a scored comparison, and document the scoring methodology so the decision can be defended if it is challenged later — by an auditor, a board member, or the vendor that lost.

The real insight is unglamorous: the vendor with the highest accuracy spec does not always win. The vendor that solves your specific problem at acceptable quality, with manageable integration and realistic costs, wins. Fit beats headline numbers, every time.

Take the next step

Want help applying this to your organisation? Use the resource below or book a 30 minute strategy call with Simon — no pitch, just practical advice.

Frequently asked questions

Related resources

Executive Resources

Find Out Where AI Can Save or Generate Money in Your Organisation

Book a free 30-minute call with Simon. Bring a real problem — staff time, governance worry, vendor proposal, failing pilot — and leave with a concrete first step you can take next week.

07973 210 895