Enterprises are growing the use of AI – but proving what it earns is the harder part

At the World Summit AI conference, Santander, Axa, ABN Amro, AMD and Albert Heijn explained how they are trying to turn AI projects into demonstrable value they can prove

A year ago, Hiek van der Scheer measured the progress of his company’s artificial intelligence (AI) projects by the number of use cases. As chief data and AI officer at Dutch bank ABN Amro, he had a target for the number of AI applications the organisation managed to get into production. The business case behind each one mattered, but not that much. Today, that’s changed.

“Now, every use case we take live has a business case,” he said, speaking at the World Summit AI conference in Amsterdam earlier in October. To get funding, an AI project has to show results in at least one of five measures the bank has set – lower costs, growth, higher customer satisfaction, better use of capital or reduced risk.

Marco Bressan, chief scientist at the Spain’s Banco Santander, has seen what happens when nobody asks for the business case. Companies hand out AI assistants and copilots, and “people do get more productive”, he said, “but they end up doing other things, or generating work for others. So, it’s very hard to justify that productivity, and all those tokens flying out of the window.” He has a name for it – the adoption trap.

Most organisations, Bressan said, promise to save so many hours: “But it’s not precise. It’s not in the books.” His bank’s way out is to move as quickly as possible from tools to AI products, such as mortgages and consumer finance offerings that use AI, where the value can be captured and shown.

Santander has committed to generating more than €1bn in business value from AI between 2026 and 2028, through extra revenue and lower costs. In the first quarter of this year, the bank reported €35m, and it expects more than €200m by year-end.

Axa has made a similar promise to its investors. Giovanni D’Aniello, group CIO of the French insurer, said Axa has been getting value from AI for more than a decade. In its Growing forward strategic plan, presented in September, the group put a figure on the next phase – €500m to €700m in annual recurring pre-tax benefits from scaling AI by 2029.

For many other organisations, such numbers are still some way off. At ING Belgium, AI now pulls company information from the Belgian Official Gazette, where firms publish changes such as new directors, and updates the bank’s systems automatically, work that used to be done by hand.

“It frees up time for other things,” said Bram Joostens, customer journey expert in business lending at ING Belgium, told Computer Weekly. “But we’ve only just rolled it out, so it’s hard to say anything concrete yet.”

Meanwhile, the next step is being built at the bank – an agent that checks incoming business loan applications for missing documents.

Forcing the numbers

What separates the organisations that can put a figure on AI from those that can’t is often who forces the question. At ABN Amro, it comes from the top. Van der Scheer doesn’t pretend measuring is easy though.

“Because the whole environment is changing, it’s sometimes hard to pinpoint whether something is really down to AI or to something else we’re doing,” he said. Even so, he works with the bank’s controllers to measure every use case against the five key metrics.

At AMD, it was cost that forced the question. When the chipmaker started deploying agents across the company, “the AI token bill took off astronomically,” chief technology officer Mark Papermaster told Computer Weekly.

His chief financial officer responded with a condition. “Look, we’ll spend the money,” Papermaster recalled her saying, “as long as we’re sure we’re getting the return on the investment.” Every group that applies AI now budgets for it in the normal budgeting process and owns proving the benefit.

Others measure application by application. “A/B testing is how medical companies have measured whether a drug works for a very long time,” said Pierre-Yves Calloc’h, former chief digital officer at French drinks group Pernod Ricard, where he spent six years deploying AI at scale.

One of those deployments gave sales reps in more than 28 countries a recommendation every Monday morning on which stores and bars to visit, and what to do once there. By comparing reps who got the recommendations with reps who didn’t, the drinks group could attribute a few million euros in benefit to it.

Pratyus Sinha, until recently senior director of responsible AI at Decathlon, starts measuring even earlier – before anything is built. When he was asked to overhaul the AI behind the sports retailer’s e-commerce platform, there were already hundreds of possible use cases on the table.

“You could just put a finger in the air and say, this one’s worth a hundred million,” he said. “But when you look at the assumptions behind that number, it quickly falls apart.” So, his team built a framework to choose between them, weighing revenue potential, risk and capacity, but above all how measurable each use case was.

Website search passed that test because its value can be calculated – missed queries multiplied by the conversion rate and the average basket value. Even so, a calculation was not enough, said Sinha. The new search agent was first tested in production with real customers, through A/B tests and a gradual roll-out, and only then scaled up.

Proof before code

Retailer Albert Heijn, a market leader among Dutch supermarkets, takes that caution further still. “We’d actually rather not use AI,” said Noortje van Genugten, the company’s vice-president of product. “If we can solve a problem more easily another way, we will.”

Its stores employ many teenagers in their first job, who are regularly asked by customers about products they have never heard of. The team suspected AI could help but wasn’t sure enough to invest. So, they added an assistant to the retailer’s in-house staff app, but initially without any AI behind it.

“My team sat in the back room of the store and answered every question themselves,” Van Genugten said. “Because we wanted to understand which questions were being asked before building anything. Maybe AI wasn’t the solution.”

Most questions turned out to be about range, stock, location in the store, price and promotions. Only then did the team build the real assistant, which still answers just that one kind of question. Albert Heijn has 125,000 staff, and the assistant now handles around 80,000 conversations a week.

Process before AI

For much of his talk, the screens beside Santander’s Bressan showed The Flintstones, the cartoon family whose stone-age life was a copy of 1960s suburbia, down to the toaster.

“Many corporations are like the writers of The Flintstones,” he said. “They want exactly the same operating model they have today, only with AI.” Many processes were designed for people to hand work to one another, he argued, and translating them one-to-one for AI misses what the technology can do.

D’Aniello sees the same reflex in insurance. Even Axa’s most modern core applications, he said, are essentially digitised versions of old paper-based processes, down to the order of the tasks and the data each step requires.

“If you’ve been an underwriter or a claims handler for 25 years, your natural behaviour when redesigning a process for agents is to replicate what you know,” he said. “But that doesn’t really work, because agentic transformation is not about automation.”

Automation, in his view, is the smallest part of the value. His first advice to staff is to forget the past and start with a blank page.

Papermaster offered AMD’s chip design process as evidence. Before agentic AI, the company found dozens of places where AI improved individual tasks. “We got 1%, 3%, 5% productivity gains,” he said. “Very helpful – but it didn’t significantly shorten the time it takes us to get new chips to market.”

That changed when models became capable enough to chain those tasks together – one agent with perhaps 10 specialised sub-agents underneath it, much like a team of experts. Applied to the whole design workflow, he said, that is now saving months on an effort that typically takes 18 months to two years and has allowed AMD to keep growing without expanding its workforce at the rate it once did.

At Pernod Ricard, the recommendations only paid off if sales reps followed them – with 10% adoption, the company would get 10% of the value. It aimed for 85%, and got there by making the job more interesting, freeing reps from spreadsheets to do the store visits and personal contact they enjoyed.

“The technology is complex, but it’s not the most complex part,” Calloc’h said.

AMD came to the same conclusion. The biggest change it needed was in culture and mindset, and every employee now has to complete AI training, while an agentic AI centre of excellence supports the rest of the company. Success stories are shared internally, and “when engineers see what another group achieved, that spurs them on”, Papermaster said.

D’Aniello made a related point about Axa: “If you don’t give all your employees access to these tools, it’s very difficult for them to imagine completely different workflows.”

Calloc’h is also candid about what he missed. His Monday morning recommendations for sales reps served a large population making many small decisions – but the biggest decisions, such as which brands to invest in or how to divide millions between countries, were still taken in the boardroom with Excel and PowerPoint. “And it’s still the case today in most companies I’ve been talking to,” he said.

An owner for every agent

As agents take on more work, the speakers kept returning to who answers for what they do.

“Even before AI, if you were accountable, you had to test the work and stand behind it,” said Papermaster. “That doesn’t change with AI.” At AMD, every agent has a manager.

Van der Scheer said of ABN Amro’s mortgage process that “we will never say: the computer says no”, with a human always taking the final decision. To keep agents in check, the bank splits them into layers. A conversational agent talks to customers or staff, a reasoning agent checks every request against the bank’s policies before passing it on, and task agents carry out the work.

That middle layer exists because an agent always wants to please, he explained – with the right questions, a conversational agent can be talked past its limits.

Axa, which operates in 52 countries, has centralised the gatekeeping. Its minimum set of guardrails was designed together with a handful of leading countries and built into modules of Axa’s agentic hub.

“Countries can do more,” D’Aniello said, “but they cannot do less.” Any business unit or claims manager that wants to start using agents has to go through central teams, which check what can be reused and make sure those guardrails are in place.

Not everyone on stage drew the line in the same place. Sinha argued that people should be able to stop an agent but not steer it, and that within clear boundaries and with a limited blast radius, it should act fully autonomously. The manager stays accountable, he said, but the action itself should not wait for a human.

Axa keeps a human both in and on the loop of every process, and D’Aniello draws a sharp line between the two. “In the loop” means deliberately leaving a share of the work with people. “What you want to avoid is that, over time, the real expertise of Axa sits in the agentic ecosystem,” he said, “and not in the heads of our specialists anymore.”

“On the loop” means experts who watch in real time how agents handle a claim, with the power to stop it, and who review afterwards what was done.

Van der Scheer sees it going the other way. ABN Amro uses agents across its software development lifecycle, with a human checking every step. But if the code is 95% right, he said, the reviewer only finds the remaining 5%, and that soon stops being an effective control.

He is convinced that within a few years nobody will review code by hand anymore, while the bank will still need to control it. “You can say there was human oversight,” he said, “but if it’s just a stamping machine, you don’t have effective control. On what human oversight really means, the jury is still out.”