To survive the AI age, the web needs a new business model

This week Hollywood released details of a new film about Sam Altman and the rise of OpenAI, whose launch of ChatGPT in late 2022 has brought high drama to the tech world. One of the most important effects of artificial intelligence (AI) so far is surely on how information is spread online. Instead of typing their questions into search engines, people increasingly pose them to chatbots.

This revolution promises to bring more knowledge more quickly to more people. Users are right to embrace it. But there is a sting in the tail. As AI-powered search engines remove the need for people to trawl the web looking for sites with answers, they are stopping the flow of traffic to those pages. Those lost visitors mean lost money. The danger is that, as answer-engines take readers away, they are removing the incentive for content to be created. The technology that is opening up the web also threatens to kill it.

Web use is hard to measure, but by one estimate monthly traffic from search engines has fallen by 15% in the past year. Some of the loudest complaints have come from the news media, an industry in which we acknowledge an interest. But the drought is a wider problem.

The cost is clear. Human traffic—monetised with ads—is the economic fuel of much of the internet. A steady flow of traffic is also needed to build online communities. 

As the old model buckles, the web is changing. It is becoming less open, as formerly ad-funded content is hidden from bots, behind paywalls. Content firms are reaching people through channels other than search, from email newsletters to social media and in-person events. They are pushing into audio and video, which are harder for AI to summarise than text. Big brands are striking content-licensing deals with AI companies. Plenty of other transactions and lawsuits are going on. Hundreds of millions of small sites—the internet’s collectively invaluable long tail—lack the clout to do this.

No one should expect the web of the future to look just as it does today. AI-powered search will rightly shake up some services: business directories, for instance, face disintermediation as answer-bots field queries such as “emergency plumber” or “houses for sale”. But the evaporation of incentives to create content presents a fundamental problem. If human traffic is drying up, the web will need a new currency.

Online innovators are trying out alternatives. Some propose a pay-as-you-crawl system, in which AI bots are charged for reading sites’ content. Others are working on systems that analyse chatbots’ answers to determine where their information came from, so that sources can be compensated. Tech firms resist such ideas: giants don’t want their crawling of the internet to be metered, and startups fear they will be made to pay for training data that pioneers like OpenAI were allowed to grab for nothing. Optimists cite the music industry, where piracy gave way to profitability when streaming platforms invented new ways to charge consumers and compensate artists.

Bringing a new business model to the web is daunting; it may take a shove from regulators to get started. Yet everyone has an interest in making content-creation pay. Publishers may be the ones complaining now, but if the content tap dries up, AI companies will suffer, too. Some are more vulnerable than others. Whereas Meta can draw on data posted to its social networks and Google owns YouTube, the world’s biggest video vault, OpenAI relies entirely on others for its content.

If nothing changes, the risk is of a modern-day tragedy of the commons. The shared resource of the open web will be over-exploited, leading to its eventual exhaustion. If that process is not stopped, one of the great common properties of humanity could be gravely diminished. The tragedy of the web would be a tragedy for everyone.

show less