‘Profiting from incuriosity’

Last May, Matthew Prince, the co-founder and CEO of Cloudflare, a major internet-infrastructure company, said his firm has tracked a dramatic shift in how tech companies interact with the web: The ratio of how many times they scrape websites to how many visitors they send back has spiked in the last year.

“Ten years ago, for every two pages that Google would crawl, they would send content creators one visitor,” Prince said. “Six months ago, the rate was six pages scraped for every one visitor. Now it’s 15 scrapes for every one visitor.” According to Prince, AI companies send exceptionally few users to external websites. “Six months ago, OpenAI was at 250 scrapes to one visitor. Today, it’s almost 1,500 scrapes to one visitor.”

For its part, OpenAI wants to replace the web as we know it. The company’s CEO, Sam Altman, said, “We think that AI represents a rare once-a-decade opportunity to rethink what a browser can be about and how to use one, and how to most productively and pleasantly use the web.”

Sites across the web are getting fewer and fewer clicks. The owners of the British newspaper The Daily Mail, for example, say their web traffic has gone into a massive slump over the past year; the number of readers clicking through to the Mail’s website from Google searches featuring AI-generated summaries has, they said, fallen by as much as 89 percent.

Google has long maintained that the web—which originally encouraged users to click through to a wide range of websites—is flourishing. But in September 2025, the company stated in a legal filing that “the open web is already in rapid decline.”

Why would that be?

Michael Socolow is a professor of communication and journalism at the University of Maine. Socolow says the web isn’t just gated off behind paywalls now; big technology companies are systematically killing it—profiting more from funneling users to specific sites, harvesting their data, and selling it to advertisers than from enabling people to browse freely. An open web, in other words, just doesn’t make enough money.

And commercial AI has just supercharged the trend. The big tech companies no longer want to serve as guides to the web; they want to keep users on their own platforms, where AI chatbots provide all the answers in one place. For all the problems with the open web, Socolow says, what’s replacing it isn’t solving them—it’s making them worse …

Gustav Jönsson: What exactly is the open web?