Why governance is the blocker, not the model
Almost no pilot dies because the model was not clever enough. They die at the point where someone has to write down what the thing is allowed to do on its own, and nobody can answer it. The capability arrived before the control surface did, so the work sits in a demo environment reading copies of real documents and producing output that a human retypes somewhere else.
That is a governance problem wearing a technology costume. The question is never “is it accurate enough” in the abstract. It is: accurate enough to do what, at what value, with whose name on the outcome, and how would we know afterwards. Until those four are answerable, no amount of benchmark improvement moves the deployment forward.
The four control surfaces
Everything practical in agent governance reduces to four surfaces. If you can name who owns each one and where it is configured, you can govern the agent. If any is missing, you are relying on the model behaving well, which is not a control.
The agent is a named thing with its own account, not a borrowed service credential. Actions attribute to it. It can be suspended by one person in under a minute.
Which systems, at which scope, enforced per request by the source system, not by a prompt asking it to be careful. Read and write are separate grants.
What it retains between tasks, where that lives, who can read it, and what happens to it when a person leaves or a contract ends.
Every decision, with the documents it rested on, at the time it was made. Reconstructable months later without asking an engineer to query a log store.
Trust as a progression, not a switch
The binary framing, automated or not, is why so many of these programmes stall. An agent that has done a task correctly two hundred times under review has earned something. An agent on its first day has not. Governance should be able to tell the difference, and the earning should be visible rather than asserted.
Learning
Reads, drafts, proposes. Takes no action. Every output is reviewed and the review is the training signal. You are measuring agreement rate, not saving time yet.
Supervised
Acts, but each action is approved before it lands. The saving is real here because the human is confirming rather than producing. Most work should live here longer than anyone expects.
Trusted
Acts alone inside a written boundary: a value ceiling, a document type, a named set of counterparties. Outside the boundary it escalates. The boundary is the grant, not the model's confidence.
The important property is that promotion is a decision a person makes on evidence, and demotion takes one click. If a level can be raised by anything other than a human with authority, such as a config change, a retry loop or text inside a document, the progression is decorative.
Designing approval and escalation
Approval queues fail in one of two directions. Too much reaches them and people rubber-stamp, which is worse than no control because it launders responsibility. Too little reaches them and the first anyone hears of a problem is from a customer.
“Confidence below 0.8” is not a policy anyone can defend. “Any new supplier, anything over $10,000, anything where the document contradicts the system of record” is.
Named individuals go on holiday. An unowned queue is where governance quietly stops. Every rule needs a second holder and a time limit after which it moves.
A queue item that shows only a recommendation trains people to click yes. One that shows the documents, the conflict and what it concluded lets them actually review, in about the same time.
The evidence you will be asked for
Assume that in eighteen months someone, whether an auditor, a regulator, a client or a court, asks why a particular thing happened on a particular Tuesday. Work backwards from that. An application log is not an answer; it tells you what the software did, not what was decided or on what basis.
What was decided, in words a non-technical reader understands.
Which documents and system records it rested on, as they were at that moment.
Which agent, at which trust level, under which job description and version of it.
Whether a person approved it, who, and what they were shown when they did.
What was escalated and not acted on: the near misses are often the more useful record.
If that record is produced as a by-product of the work, you have it. If it has to be assembled on request, you do not have it, you have a project to build it during an incident.
Six ways this goes wrong
Silent scope creep
An integration is added for one task and quietly widens what the agent can reach everywhere else. Permission changes need to be reviewable events.
Shared credentials
The agent runs as a generic service account, so its actions are indistinguishable from everyone else's. Attribution is gone before you start.
Orphaned agents
The person who commissioned it changed role. It is still running, still trusted, and nobody is reviewing its output.
Instruction in content
A document tells the agent what to do and it complies. Not fully solvable, but the floors it cannot cross must be outside the model's reach.
Approval theatre
Ninety-eight percent approval rate is not evidence of quality, it is evidence that nobody is reading. Sample the approvals separately.
Memory with no lifecycle
Useful context accumulates and is never scoped, expired or deleted, so a subject access request becomes an archaeology exercise.
See it against your own boundary
Bring one process and the rule you would want enforced on it. Thirty minutes is enough to find out whether the control surfaces line up with how you actually work.