What can AI actually deliver?
Organisations trying to judge whether AI is doing useful work for them have been handed two easy measures over the past year, one that rates a system by the crowd its demonstration draws, and another that rates employees by how much AI they use. Both measures are simple to record and compare, yet neither shows whether the technology has delivered any value once it is applied to everyday work.
Recent events have exposed the limits of both measures, beginning with a humanoid robot that appeared to read a man's mind in front of a crowd of interested attendees at a technology conference, and continuing with reports that engineers at some of the largest technology companies were being ranked by how many AI tokens they consumed. Neither case measured AI by the work it does, a distinction worth making in an enterprise market where many organisations are still at an early stage of adoption, with one study finding that 55% of French respondents were operating in proof-of-concept mode and only 18% had a clear roadmap to full deployment. Speakers at that conference, VivaTech, focused more on what AI can deliver today than on what it might do next.
The first case was designed to capture attention, with French neuroscience start-up HABS and robotics distributor Innov8 Group fitting a volunteer with a sensor-equipped headband that reads brain activity before asking him to choose one of three cards and concentrate on it. The robot then pointed to the card he was thinking about, producing the kind of immediate impact that makes a technology demonstration memorable, even if the crowd’s reaction says little about what the system can reliably do beyond that moment. With three cards on the table, a random guess would have been right a third of the time.
The second case raised a different question about how AI is being measured, with reporting exposing a practice nicknamed tokenmaxxing, in which engineers at firms including Meta and Microsoft were ranked on internal leaderboards by how many AI tokens they used. The Meta leaderboard, built by an employee, came down within days of being reported, and in September Meta told engineers that token usage would no longer count towards their performance reviews, highlighting the problem with treating usage as a measure of success when using more of a resource does not necessarily mean producing more value. That excess consumption also carries an environmental cost, since every AI query draws on the energy and water used to power and cool the data centres behind AI services, even as providers report drastic falls in energy use per prompt.
The consistency gap
Consistency over longer tasks offers a useful test of the gap between what AI tools promise and what they deliver in everyday use. Large language models can still struggle to carry instructions consistently through longer pieces of work, with one study finding an average 39% drop in performance when instructions were spread across several turns, leaving users to repeat the same guidance and steer the process themselves. Productivity can make that gap harder to ignore, because in some settings using AI can even cost time, with one small METR trial of 16 experienced developers using early-2025 tools finding that they took 19% longer to complete tasks.
Neither limitation necessarily stops organisations from using AI, and both may improve, but together they help explain why enterprise adoption has been more measured and selective than the promises of recent years implied. The same uncertainty around value can also encourage organisations to measure activity instead, which helps explain the appeal of tokenmaxxing, where token counts provide a simple metric that can be compared across employees.
Proof follows deployment
Arthur C. Clarke wrote that any sufficiently advanced technology is indistinguishable from magic, and the HABS demonstration made that comparison hard to avoid. But magic is judged by the effect it creates, while useful technology has to be judged by what it can reliably do, making an impressive demonstration a poor measure of whether the underlying system represents a genuine advance.
HABS is developing the technology for applications that extend beyond the demonstration, and says its system relies on a continuously enriched language model for brain-wave data that can detect neuromarkers including pain and fatigue. The demonstration showed that the headband could detect a clear signal under controlled conditions, but it did not show whether that capability would hold in the healthcare and cybersecurity settings where HABS says its technology is already used, where reliability, scale and cost would determine whether the technology could be used in practice. The actual test of the technology begins in the gap between those two settings.
Staged demonstrations and token leaderboards reveal the same weakness, measuring something that is easy to count rather than the harder evidence of whether an AI system delivers useful results. The meaningful measure is what the technology can solve once it leaves controlled conditions and has to perform actual work. Organisations need that evidence before committing significant budgets, while Europe needs it as it weighs the case for investing in its own AI capacity, a question that dominated VivaTech’s debates on digital sovereignty. For technology companies, that makes evidence from real deployments a more persuasive story than any demonstration or usage figure.
If models improve on longer tasks and costs keep falling, the gap between what AI promises and what it delivers may narrow, but the more credible test will be whether organisations are still paying for these systems a year after their pilots end. Demonstrations can show what is possible, but the evidence will come from whether AI continues to earn a place in everyday work.