Soundings
Our Perspectives and Insights
You Can Mandate AI. You Can't Mandate Trust.
We have been covering the journey of an AI prototype: completing the prototype, building its production readiness, and now putting it in the hands of users.
That last step is where most of them quietly die. It does not happen in the steering committee that approved the budget. It happens at one desk, in one person’s head, and it can undo the other two phases after you have already paid for them.
Once we got that grant and clinical trial administration software production ready, one small issue almost lost the entire adoption curve. With the technology at the time we had separate server-side and frontend calculation scripts, and they had drifted. On a budget screen, when you edited the numbers and clicked save, the total changed by two cents. For users who spend all their working hours buried in Excel, watching numbers change by themselves caused instant distrust in the system, and it took months of validation proofs and presentations to win it back.
That was deterministic software with a bug in it. The numbers were supposed to match, one day they did not, and it cost months. AI is not a system that occasionally drifts by two cents. It is a system built to give a different answer, by design.
AI Changed One Number in the Adoption Calculation.
Every user runs the same private arithmetic: is the benefit to me bigger than the cost to me? Not cost to the company. Cost to me, this week, at my desk.
With deterministic software, the cost of adoption was learning it once. You learned the screen, and after that, the same input gave the same output forever. AI does not work that way, so the cost is not learning it once. It is checking it every time.
Nobody ran a change management program for turn-by-turn navigation, because checking it was free: look up, is that the road? Now take the AI tool you just rolled out. It drafts the analysis, prices the deal, summarizes the case file. To know whether it is right, you have to do the work. Verification costs as much as the work, and that is a recurring tax you levied on the person with the least ability to refuse it.
Verification costs as much as the work.
One Wrong Answer Is a Bug. A Pattern Is a Verdict.
Your user is not being handed an answer. They are being handed a draw from a distribution. Most of it is good. Some of it is not, and nobody can tell them in advance which today is.
They can live with that, and they do it everywhere else in their lives, but only on two conditions. They have to be able to say this one was wrong. And saying so has to change something.
A single bad output that gets reported, fixed and never returns is not a trust problem. It is a bug, and users have forgiven bugs since software existed. What breaks trust is accumulation. It is four o’clock on a Friday, the thing is due, and the tool is wrong again in the same way it was wrong on Tuesday, and nothing they said about Tuesday made any difference. That is the moment adoption silently dies.
So the user does the only rational thing left. They verify everything, which costs them time, or they quietly do the work themselves and use the tool as a formality. Either way the time savings you promised is gone, and the time savings was the entire pitch.
Either way the time savings you promised is gone, and the time savings was the entire pitch.
Non-Adoption Is a Readout, Not Resistance
Donella Meadows put it plainly: We can’t impose our will upon a system. She was right, and most rollouts try it on the organization anyway. But a system does not only refuse. It answers, and yours has been answering you all along.
That distinction matters because it is the difference between a problem you can act on and a problem you can only complain about. Resistance is a story about your people. A readout is a measurement of your system, and you can change a system.
If the tool is wrong in the same way twice and there is nowhere to report it, that is not resistance. If one person now checks the machine’s homework while the savings land on somebody else’s slide, that is not resistance. If the people who went first got nothing for going first, that is not resistance.
That is your own system, returning an honest answer to a question you asked badly.
The Four Questions Your User Is Actually Asking
You cannot argue anyone out of their own math. You can change the inputs, and for AI there are four.
- How often is it wrong, and in what way? Not does it work. They are reading a distribution, so give them the real error rate on their own work before they find it themselves. A known failure mode is survivable; an unknown one is not.
- Can I check it faster than I could just do it? If not, you have added work. Citations, source rows, a visible diff against the original: anything that makes a spot check cheaper than the task itself.
- When it is wrong, can I make it stop? This is the one most rollouts skip. Give them a way to flag a bad output, then show them something actually changed. A feedback path nobody answers teaches the user that the bad draws are permanent.
- Why are we doing this, and what happens to me when it is wrong? Ship the reason with the change, not a month later. Then say out loud who owns the error, because a user signing their name to an output they cannot check has taken on the risk without the authority.
The Bottom Line
You can buy the model. You can mandate the login. You cannot mandate the trust, because trust is not a policy. Trust is a verdict, and your users re-issue it every time they look at an output and decide whether they have to check it.
Trust gets won the way it has always been won: by being right, visibly, on the boring work, over and over, and by making it cheap to catch you when you are not. That is slow, and there is no memo that does it faster. Skip it and you will have bought something that works exactly as designed and changes nothing.
A Pilot & Rutter principle.