First, work out what you are actually buying
Three quite different things are being sold under similar language, and the evaluation criteria for each are not the same. Deciding which one you need before you take the first call saves a quarter.
Answer questions, draft text. A person carries the output somewhere else. Judge on answer quality and adoption. No governance burden because nothing acts.
Deterministic flows with a model in one step. Judge on reliability and maintenance cost. Breaks when the input varies, which in document work it always does.
Own a piece of work end to end, hold context between tasks, act inside a granted boundary. Judge on governance, memory and evidence. Accuracy is table stakes.
If what you need is the first, most of this guide is overkill and you should buy on usability. It is written for the third, where a wrong choice is expensive to unwind because the thing has been given permissions.
Six criteria that decide the outcome
Governance
Are autonomy boundaries configured by you, in business terms, and enforced outside the model? Can you demote in one click? This is the criterion that kills deployments.
Memory
Does it retain anything between tasks, or does every conversation start from nothing? Retrieval over a document store is not memory of your organisation.
Integration depth
Read-only or read and write? Enforcing the source system's permissions per request, or holding one admin token? The second is a finding waiting to be written up.
Decision record
Produced automatically, readable by a non-engineer, and reconstructable months later, or an event log you would have to interpret during an incident?
Security posture
Where it runs, who the subprocessors are, what reaches a model provider, whether your data trains anything. Ask for the list, not the badge.
Failure behaviour
What happens when it does not know? Stops and asks, guesses confidently, or silently drops the item. Ask to see it fail in the demo.
Questions to put to every vendor
Phrased so that a vague answer is visible as a vague answer. If two vendors give you the same words, ask the follow-up in brackets.
In the product, not in a slide. If it is a prompt field, the boundary is advice rather than a control.
Live, in their own demo tenant. How long it takes them tells you what your audit will cost.
A straight list is a good sign. “We are SOC 2 ready” means nothing and should be treated as a no.
If the answer needs a follow-up email, the list is not being maintained.
Bring one. Scanned, slightly wrong, missing a field. That is what your inbox looks like.
Exportable in a usable form, or a lock-in you are choosing to accept with your eyes open.
Commercial models, and what each one costs you later
Predictable, but penalises exactly the outcome you want: the work being done by fewer people. Watch for viewer seats needed just to approve.
Aligns cost with value, and makes budgeting hard. Ask what counts as an action and whether a retry counts twice.
Easy to reason about against a salary line. Check whether a second process means a second licence.
Usually hides the real number in implementation services. Ask for the total first-year figure including people.
In all four cases ask the same two questions: what does year two look like at three times the volume, and what is the cost of stopping. Our own answer is on the pricing page, including the parts that are not a fixed number yet.
Designing a pilot that proves something
Most pilots are designed to succeed, which is why so many of them end without a decision. A useful one is designed to be capable of failing on a criterion you agreed in advance.
One process
High volume, real documents, a clear owner, and a definition of correct that two people would agree on. Not three processes.
Real mess
Include the scanned ones, the exceptions, the supplier who emails a photo. A clean sample tells you nothing you need.
Baseline first
Measure today's time per item, error rate and exception rate for two weeks before anything is switched on. Without it there is no result.
Start supervised
Agreement rate under review is the metric that matters in weeks one to four. Autonomy is a decision you take on that evidence, not a pilot setting.
Written exit
The numbers that mean yes and the numbers that mean no, signed by the process owner and the vendor before day one.
Ninety days
Long enough for the exceptions to show up, short enough that the sponsor is still in the role when it ends.
Red flags
An accuracy percentage with no dataset, no definition of correct, and no failure mode attached.
“Fully autonomous” offered as a feature rather than as something you grant deliberately.
A claim that prompt injection has been solved. Nobody can know that.
A demo on synthetic documents, with reluctance to try yours in the room.
Integrations described as “on the roadmap” for the system your process actually depends on.
No named reference doing the same process at a comparable size, at any stage of the conversation.
Bring the list. We'll answer it in order.
Including the questions we come out badly on. If we are the wrong fit for where you are, a short call is the cheapest way for both of us to find that out.