Open any newsletter this month and you will read that artificial intelligence makes people 25% faster. Sit through a vendor demo and the number climbs higher. If you run a small business or lead a team, it is a tempting promise: buy the tool, hand it around, watch the work fly out the door.
The trouble is that the research behind those headlines is far more interesting – and more honest – than the marketing. The best studies do not actually agree with each other. And the gap between them is exactly where a smart owner can make a good decision. Here is what the evidence really shows, and how to test it on your own team without betting the quarter on hype.
Where the 25% number comes from
The famous figure traces back to a large field experiment by researchers at Harvard Business School and Boston Consulting Group, published under the title Navigating the Jagged Technological Frontier. They gave 758 consultants access to GPT-4 on realistic business tasks. Compared with colleagues who had no AI, the assisted group completed about 12% more tasks, worked roughly 25% faster, and produced work rated meaningfully higher in quality.
Those are real, well-measured gains, and they are worth taking seriously. But notice the fine print: this was a specific kind of work – idea generation, drafting, structured analysis – that sits comfortably inside what today’s models do well. When the task fit the tool, the tool was genuinely excellent.
The other half of the same study
The reason the researchers called it a jagged frontier is that the same study found a sharp downside. On a task deliberately chosen to sit just outside the model’s reliable range – one that looked similar but required judgment the AI could not supply – consultants using AI were noticeably more likely to land on the wrong answer than those working without it.
Read that twice, because it is the whole game. The AI did not announce which side of the line a task was on. It answered every question with the same confidence. People who trusted it uniformly did better on the easy-fit work and worse on the tricky work. The tool did not just add speed; it added speed in a direction you could not see without checking.
When AI made experts slower
If the jagged frontier were the only wrinkle, the advice would be simple. But a separate, carefully run trial complicated the picture again. When experienced open-source software developers used AI coding assistants on their own familiar projects, they were about 19% slower than when they worked without them – and, tellingly, they believed they had been faster.
That last detail matters more than the number. The developers felt more productive while measurably losing time to reviewing, correcting, and re-prompting. Perceived speed and actual speed came apart. For a busy owner running on gut feel, that is a warning: “it feels faster” is not evidence, and it can point the wrong way.
Why team-level gains are smaller than the demo
Here is the finding that should shape your budget. When researchers stepped back from individual tasks and reviewed several independent studies of whole teams and companies, the productivity gains converged on something closer to 10% – not the 25% you see on a single task in a lab.
Why the shrinkage? Because a business is not a stack of isolated tasks. Work has to be handed off, reviewed, corrected, and coordinated. An individual who drafts a proposal twice as fast still waits on the same approval, the same client reply, the same teammate downstream. AI speeds up one step; the rest of the flow does not automatically keep up. As BCG put it in a follow-up analysis, the bigger opportunity is not making individuals faster but changing how work moves across the team – and that is a management problem, not a purchase.
What this means for a small team
Put the studies together and a practical picture emerges:
- AI is genuinely strong on well-scoped, draft-and-refine work – first drafts, summaries, brainstorming, reformatting, boilerplate. Lean into these.
- It is quietly risky on judgment calls that look routine but are not. Keep a human decision at those points.
- Feeling faster is not being faster. If you cannot measure it, you do not know it.
- The whole-team payoff comes from smoothing the flow, not from any single person sprinting.
None of that means “skip AI.” It means adopt it the way you would any other change to how work flows – deliberately, and with a way to see whether it actually helped.
How to test AI on your own workflow
You do not need a research budget to run a fair trial. You need two weeks and a little discipline:
- Pick one recurring task your team already does often – writing proposals, first-pass support replies, meeting notes. Pick something you can count.
- Write down today’s baseline before you add anything: roughly how long it takes and how often it needs rework.
- Add the tool to just that task for two weeks. Change one thing at a time so you know what caused what.
- Watch the whole flow, not the moment. A task that is drafted in half the time but bounces back twice in review is not a win. Track how long work takes from start to truly finished, and how often it comes back.
- Keep a human check on the judgment steps – anything where a confident wrong answer would cost you a client or a correction.
A kanban board makes this almost automatic. Move the trial task across your columns as usual and you will see, in plain sight, whether cards are finishing sooner or just piling up in review. The board turns “it feels faster” into something you can actually look at.
The takeaway
The evidence is not that AI productivity tools are overhyped, and it is not that they are magic. It is that they are uneven – powerful on the right work, misleading on the wrong work, and only as good for the whole team as the workflow you drop them into. The owners who win with AI are not the ones who buy first. They are the ones who run a small, honest test, keep judgment where it belongs, and fix the flow around the tool.
Start with one task, watch it move across your board, and let the finished-work column tell you the truth.
Sources & further reading:
- Navigating the Jagged Technological Frontier (Organization Science)
- AI at Work: Why Strategy Matters More Than Tools (BCG)
- The best evidence on AI and productivity does not agree with itself (Silicon Canals)