Soundings
Our Perspectives and Insights
Most AI Pilots Aren’t Killed. They’re Never Allowed to Die.
Here is the number everyone repeats: for every 33 AI pilots a company starts, only four reach production, IDC’s count in Lenovo’s 2025 CIO Playbook. It gets passed around as a scandal, but it is really two questions wearing one answer: should every pilot reach production, and for the ones that should, why do so many not? This piece takes the first. The next takes the second.
Start with the first, because almost nobody does. A pilot that reaches its end date is not a pilot that worked; it is a pilot that ended. These pilots rarely die because the tool cannot do the job. They die because of how they were run: no honest read, week to week, on whether it was working, and no willingness to stop the ones that were not.
I see it with clients now. They ask me to help get their AI pilot off the ground, and it turns out the task they want done needs the same answer every single time. Once they confirm that, I tell them to kill the AI pilot and go write an automation script instead.
The Sunk Cost Becomes the Baby
A company puts real money and real months into a pilot over time. Somewhere along the way it stops being an experiment and becomes a baby. People are attached to it. Budgets have been defended in its name. Someone’s reputation is riding on a good outcome. And the question in the room quietly changes from “is this working?” to “how do we make sure something ships?”
That is sunk cost. The money and time already spent are gone whether you continue or stop, but they bend every decision that comes after toward continuing. Add the honest human hope that all that effort has to add up to something, and you get a dead pilot walking, escorted politely into production by people who cannot bring themselves to call the time of death.
MD Anderson lived this one in public. In 2012 the cancer center hired IBM to build an AI advisor for its oncologists, a six-month job scoped at 2.4 million dollars. It was extended twelve times. By the end IBM had been paid about 39 million dollars, the consultants roughly 23 million more, and the total ran near 62 million for a system that never connected to the hospital’s medical records and was ultimately declared not ready for clinical use. Nobody set out to spend 62 million dollars. Each extension was simply easier to sign than the decision to stop, because stopping meant admitting the last one had been wasted. That is how a pilot becomes a baby: not one bad call, but a long line of reasonable continuations nobody was willing to refuse.
Agile Was Always About Being Willing to Stop
The fix is not a fancier or new process. It is an old idea most companies say they adopted and never actually did: fail fast, change direction. Agile, underneath the stand-ups and the sprint boards, is a stance. Put something small in front of reality, look honestly at what reality says, and be willing to change it or walk away. That willingness is the entire point, and it is the exact part organizations quietly drop. They kept the ceremonies and threw out the reality check (anyone hear of “agile-fall”?).
They kept the ceremonies and threw out the reality check.
A pilot is reactive and fast-moving by nature. That is a feature, not a mess. It also means validation cannot wait for the end, because the thing you are testing keeps changing underneath you. You need a real read at every step and a course correction at every step. Reaching the original deadline only tells you the calendar moved. It says nothing about whether you built something worth keeping.
This is not new, and Jeff Bezos said it to shareholders in 2016:
“Failure and invention are inseparable twins.”
– Jeff Bezos
Amazon shows the other side of the same coin. It has a long habit of pulling the plug on projects that are not working, quickly and without much ceremony, and that willingness to stop is exactly what freed it from initiatives like the Fire Phone to instead place the bets that turned into Alexa and AWS. Being willing to stop is not the enemy of ambition. It is what makes ambition affordable.
How to Run an AI Pilot You’re Willing to Kill
A good pilot is designed to give you a real answer, and the deck should be stacked toward no. Make it hard to say yes. Put the burden of proof on continuing, not on stopping, and you force the honest validation a pilot is supposed to produce.
- Write the kill criteria before you start. Decide, in advance and in writing, what result would make you walk away. A bar you set after you are attached is not a bar.
- Validate at every step, not at the finish. Put up checkpoints the pilot has to clear to earn the next one. Something that cannot show progress by week three does not get to week ten on faith.
- Separate the sunk cost from the decision. Say it out loud in the room: that money is gone either way. The only question is whether the next dollar is worth spending, never whether the last one was.
- Make killing it the default. The pilot does not continue unless someone can argue, with evidence, that it should. Flip the burden: you do not justify stopping, you justify going on.
- Refuse to count “we finished” as “it worked.” Hitting the deadline is not a result. A result is an outcome you can measure and point to. No number, no promotion to production.
Make killing it the default. The pilot does not continue unless someone can argue it should.
The Bottom Line
A pilot is a hypothesis, not a down payment. The whole reason to run one is to find out, and “find out” has to leave room for the answer to be no. The pilots actually stuck in that 88 percent were not beaten by the technology. They were beaten by their own unwillingness to look at a struggling pilot and say “no” early, while it was still cheap to say. Run the pilot like you actually mean the question. Then be brave enough to hear the answer. And if the answer is “no”, that allows you to move to the next hypothesis, which got Amazon Alexa and AWS.
And when a pilot does earn its way through, the job is not over, it changes shape. Getting something from zero to one mile an hour is a different skill than getting it from one to sixty, and that second one is organizational, not technical. That is the second question this number hides, and it is the next piece.
A Pilot & Rutter principle.