Everyone Is Buying Tools. Almost Nobody Is Building Practice.

A company I worked with earlier this year had spent somewhere north of forty thousand euros on AI licences before anyone asked what the tools were supposed to change, and when we finally sat down and listed the work they wanted moved, it turned out that most of it had never been written down anywhere, which meant there was nothing for the model to be good or bad at. That is not an unusual story. It is close to the median one.

The gap that produces it has been open for three years now, and it is worth understanding precisely, because the industry keeps describing it as a technology problem when it has become something else.

Three years, three different conversations

In 2024 this was a promise, and mostly an untested one. The demos were staged, the pilots quietly died, and the people insisting that everything had changed were describing a capability that did not exist yet. They were right about the direction and badly wrong about the date, which is the most expensive kind of right.

In 2025 it started being true. When Anthropic shipped Opus 4.5 last November the change was not that the model got smarter in some abstract sense, it was that these systems began finishing multi-step work without a human rescuing them halfway through, and finishing is the part that turns a demo into a tool.

In 2026 the capability went mainstream while the practice stayed nowhere. Claude Cowork moved from research preview in January to generally available in April, Perplexity put an agentic machine on people's laptops, Linear shipped an agent into the board your team already works from, and Apple is rebuilding Siri around the same premise. Buying access to all of that now takes about four minutes and a corporate card. Knowing what to do with it takes considerably longer, and nobody is selling that part.

The most useful experiment anyone has run

Last year METR ran a randomised trial on sixteen experienced open-source developers, working in repositories they knew intimately, on real issues from their own backlogs. Before starting, those developers forecast that AI assistance would make them roughly a quarter faster. Afterwards, having lived through it, they reported that it had made them about a fifth faster. The measured result was that it made them nineteen percent slower.

I want to be careful with that study, because it gets quoted far more broadly than it deserves. Sixteen developers, working in codebases they had memorised, on open-source issues, using early-2025 tooling that METR has since superseded with a continuation pointing the other way. It is not a law about knowledge work, and anyone telling you it proves AI does not work is selling something too.

The part that does generalise is not the nineteen percent. It is the forty-point gap between what those developers experienced and what actually happened. They had just done the work, they had every reason to know, and they were confidently wrong in their own favour. Intuition, in this domain, is not evidence.

Klarna ran the same experiment with a billion euros behind it

In February 2024, Klarna announced that its AI assistant was doing the work of 700 full-time customer service agents. Two-thirds of all chats, 2.3 million conversations in the first month, average resolution time down from eleven minutes to under two, and a projected forty million dollars in profit improvement. It was the most-cited AI success story in Europe for about a year.

By May 2025 they were rehiring humans. Sebastian Siemiatkowski, their CEO, put it plainly: "We focused too much on cost. The result was lower quality."

What went wrong is more interesting than the reversal. The AI was genuinely good at the high-volume, repetitive two-thirds. It struggled with the long tail, the complicated and emotionally loaded cases where somebody needed judgment rather than a fast answer. And crucially, the metrics they had chosen could not see that failure. Resolution time went down. Volume went up. Both numbers looked excellent while the experience of the customers who mattered most quietly got worse. They eventually fixed it, not by abandoning the AI, but by changing what they measured and putting a human back in reach.

That is the same finding as METR, at company scale and with real money attached. The tool was not the problem. The absence of any honest measurement of quality was.

It happened to me too

I am not writing this from the outside. When I first got properly fast with these tools my output jumped, and within a couple of weeks I could feel the quality starting to drift behind it, which is the trade almost nobody catches in time because output is loud and quality is quiet. So I pulled the pace back on purpose and rebuilt the habit, treating the model as something that accelerates work I still own rather than something that does the work instead of me. Noticing that drift early is most of the skill, and it is the first thing I look for now in every team I train, because by the time it shows up in the output it has usually been there for a while.

At Vandall we built a cold outreach engine, and it taught the same lesson in miniature. The first version could write. What it could not do was know when a follow-up was warranted, so it produced perfectly fluent messages at moments that made no sense, and a fluent message at the wrong moment is worse than silence. If we had shipped that iteration, a lead's first experience of us would have been obviously generated text instead of a considered introduction, and you do not get a second first impression.

So we trained it, gradually, against what a good message from us actually looks like. It is genuinely useful now. It also still has a human in the loop for anything non-deterministic, and I expect it always will, because the cost of one bad first touch is far higher than the cost of thirty seconds of review.

The pattern underneath all three stories is the same. AI is not a solution in itself. It works when you point it at a specific workflow, stay close to it, correct it, and grow it into something that meets your standard. It fails when you plug it in and walk away, because the thing it does most reliably is produce plausible output, and plausible is exactly what you cannot audit at a glance.

What is still genuinely unsolved

We are further along than we were in January, and it is worth being fair about that. Memory has converged on a sensible three-layer shape, and the better teams have stopped asking a model to decide every step, wrapping the paths that matter in plain deterministic logic instead.

The gaps that remain are structural rather than cosmetic. Agents still act on stale memories and nobody has a clean answer for how a system should forget, which sounds academic until an agent confidently applies a policy you retired in March. When something goes wrong deep inside a chain of twenty steps, working out why is closer to archaeology than debugging, because the reasoning that produced the error was never written down anywhere you can inspect. And access rights are the weakest link by a distance, because every one of these systems was designed to be helpful long before anyone designed it to be careful.

None of that means wait. It means the practice you build around the tool is currently doing more work than the tool is.

The install-everything trap

The way most people close that gap is to borrow someone else's practice. Someone you respect publishes their agent setup, you install it, and your configuration gets a little better and a little less yours. Do that thirty times across a team and the costs arrive quietly, because your context is now full of other people's preferences, your output has drifted toward whatever the popular configuration produces, and you have inherited a set of judgment calls that nobody at your company ever made.

Then there is the part almost nobody is watching.

Those figures come from a security analysis of publicly listed agent skills published this year. The detail I would sit with is the last one: eight confirmed malicious skills were still downloadable on the day the researchers published their findings.

The mechanism is worse than a bad file slipping through review. In June a security team built a plausible landing-page skill, published it to a well-regarded repository, and watched it reach around twenty-six thousand agents, some of them on corporate accounts. It passed every scanner it met, because at the moment of scanning it was completely clean. Then, after approval, they repointed the external link the skill depended on. Their payload only harvested email addresses, since the point was to prove it could be done, but the link could have pointed anywhere.

You cannot scan your way out of that, because an instruction that arrives after the check is an instruction the check never saw. The engineering controls are real and unexotic, and you should apply all of them: pin your versions, vendor anything that fetches at runtime, scope what each agent is allowed to reach, allowlist what it can talk to. What none of that decides is which agent should have been anywhere near the customer database to begin with, and that is a judgment call about your own business that no tool can make for you.

Why I believe data centre expansion will slow down

There is a version of this argument that says none of it matters, because compute keeps getting cheaper and the frontier keeps moving, so you should simply rent the best model available and stop worrying.

In the 1890s, London ran on roughly three hundred thousand horses. Each one produced something like fifteen kilos of manure a day, and the city was drowning in it. The streets stank, the flies carried disease, and the projections were genuinely alarming: a widely repeated estimate of the period had London buried nine feet deep within fifty years. An international conference convened in New York in 1898 to solve urban manure and broke up after three days without an answer, because within the terms of the problem there was no answer. Then the internal combustion engine arrived, and by 1912 the horses were largely gone. The crisis was never solved. It was made irrelevant by something that was not part of the debate.

The AI build-out has the same shape. Capital spending on capacity is running at close to double last year, and yet grid connections in the major markets now take four to five years, data centre electricity demand is forecast to grow another twenty-six percent this year, and McKinsey is now publishing on the risk of overbuilding. When the binding constraint stops being ambition and becomes the electricity supply, the pressure to do the same work on dramatically less hardware becomes enormous, and that pressure has historically been very good at finding an answer.

I am not going to tell you local models will be as good as the frontier by a specific date, because I do not believe that and the people saying it are guessing. What I will say is that building your entire operating model on the assumption that you will rent someone else's computer forever, at whatever price they choose to set later, is a strange bet for anyone with a five-year horizon.

What actually compounds

The road worth walking is slower than installing things, and it does not photograph well.

At Vandall we run agents against our own board and our own codebase, and the useful part was never the agent. It was writing down how we actually work, in enough detail that something else could execute it, which turned out to be a far harder and more valuable exercise than anyone expected. I have built the same layer for clients, and every single time the AI was the easy half. The real work was sitting down with the people who do the job and writing their standards into the system, so that what comes out the other end is recognisably theirs rather than the internet's.

Some of that system will be AI. A surprising amount of it should be plain, well-designed, deterministic process, the kind that has worked for forty years and does not hallucinate and does not need a licence renewed. Knowing which half is which is most of the job, and it is the part that survives every model release.

If that sounds like your situation, book a call and we will go through where your team actually is, which decisions you have quietly handed to a model without meaning to, and what is worth building rather than buying.

Sources

• Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (METR)
• We Are Changing Our Developer Productivity Experiment Design (METR, February 2026)
• Exploring the Emerging Threats of the Agent Skill Ecosystem (arXiv)
• The Story of Skills: how we hijacked 26,000 agents with one Instagram ad (AIR)
• SKILL.md and the new AI supply chain attack surface (Cloud Security Alliance Labs)
• Klarna reinvests in human talent for customer service after AI chatbot push (Customer Experience Dive)
• The AI Productivity Paradox: telemetry from 10,000+ developers (Faros AI)
• The AI adoption puzzle: why usage is up but impact is not (BCG)
• Claude release notes, including Cowork (Anthropic)
• Introducing Linear Agent (Linear, March 2026)
• Personal Computer is here (Perplexity)
• Apple goes agentic: welcome to the new Siri (Forbes)
• Powering AI: how real is the risk of overbuilding? (McKinsey)
• Data centre electricity demand to grow 26% in 2026 (Gartner)