Soundings
Our Perspectives and Insights
Your AI Needs a QA Line, Not a Green Checkmark
The demo went well. The AI agent answered every question the team threw at it, pulled the right records, and wrote a clean summary at the end. Then the most senior person in the room leaned back and asked the question that mattered: “How do we know it keeps doing that when nobody is watching?”
Nobody answered. Not because the team was hiding anything, but because nobody in the room knew what a true answer looked like.
An engineering team that has only ever shipped regular software has two honest answers to that question. We do not know. Or it just will, which is false confidence. The first answer kills the budget. The second one kills trust the first time the agent gets something wrong in front of a customer.
Either way the pilot stalls, and it stalls on the question and not knowing how to answer it. Technology is not the issue.
The Difference Between Verification and Assurance
Verification is what you do to deterministic software. You prove it correct once, the tests go green, and the checkmark holds until somebody changes the code. Assurance is what you do to a system whose output varies. You sample it, set a threshold for acceptable performance, name who accepts the risk that is left over, and check again when something changes. Nothing ever goes permanently green.
AI belongs in the second bucket. Most companies are still trying to put it in the first, and then wondering why the answer never comes.
We Have Solved This Problem Multiple Times Before
No distribution center has ever proven every shipment right. They agreed on an error rate, and somebody owns it. A carton gets to QA one of two ways. It gets flagged, because a person noticed a bad seal, or the scale says it weighs more or less than it should, or an RFID tag inside is wrong or missing. Or it gets pulled at random: a fixed 3 percent comes off the line no matter what, because every detector misses things. QA never knows which route a carton came from, so every one gets the same full checklist.
Your credit card works the same way. The issuer does not verify every charge. It scores each one against your pattern, holds the ones that look wrong, and counts on you to flag the ones it misses. Nobody calls that broken. It is a threshold somebody chose, tuned between declining your dinner and letting a thief through.
W. Edwards Deming said it best: “Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.” Build quality into the product: better prompts, better data, tighter guardrails. But part of your line is a model you rent, and it will answer the same question differently tomorrow. You can build quality in. You cannot build the variation out. So the sample never goes away.
You can build quality in. You cannot build the variation out. So the sample never goes away.
Four Things to Put in Place Before Anyone Asks How You Know
- A flag route. Give people a one-click way to flag a bad answer, and add detectors that know what a good one looks like: format checks, policy checks, an answer that cites nothing.
- A random pull. Sample a fixed percentage of everything, flagged or not. Your number depends on what a miss costs you. The job is catching what your flags let through, and watching the trend.
- One checklist, applied blind. Reviewers should not know whether an answer was flagged or pulled at random, or the ones that seem fine get a lighter look.
- A number, a name, and a trigger. Agree on the error rate you will accept, put a person’s name next to it, and decide what forces a fresh look: a new model version, a prompt change, or a climbing error rate. Expect the upgrade to trip your detectors even when the new answers are better. A detector can tell you something changed. Only a person can tell you whether it changed for the better.
A detector can tell you something changed. Only a person can tell you whether it changed for the better.
The Bottom Line
“How do we know it keeps working?” is the right question. The mistake is expecting a checkmark as the answer. The real answer is what gets flagged, what gets pulled, what error rate you will live with, and whose name is on it. That’s not a one-time system test. It’s a recurring process.
Nobody checks every carton. They just make sure somebody is always opening one.
A Pilot & Rutter principle.