The seismic impact of AI on web economy and what publishers and brands can do
Dave Bouskill and Debra Corbeil, creators of the renowned travel blog The Planet D, reported a 90% drop in traffic during Gemini’s initial rollout. The couple had to lay off staff and ultimately shut down their website. “I feel betrayed,” Bouskill said. “We built something for 16 years, and it was gone in six months.”
This is just one anecdote but no matter how you look at the data, growth of Artificial Intelligence (AI) driven search and AI agents is upending the web’s economy and business model as we know today, and with it degrading the thriving ecosystem of content development and free distribution on the internet.
Search engine crawls websites >> User searches for information>> SERP ( Search Engine results page) sends user to website>> Website shows ads and earns revenue
- AI-driven search is causing a substantial average traffic decline of 20–60%. ecommercegermany.com, emarketer.com
- AI crawler activity is significantly outpacing human visits, with crawl-to-visit ratios soaring from 2:1 for search crawlers to 250:1 and higher for AI crawlers. Matthew Prince, Cloudflare CEO
- For the first time, bot traffic, largely AI-based, exceeded human traffic, constituting over 51% of total web traffic. Infosecurity Magazine
Coupled with 2024 IAB PwC revenue report – stating that 80.8% ad revenue is shared by the top 10 companies (e.g. Google, Meta) and 11.0% by the top 11–25 (social media, streaming, ecommerce), the trend is a clear signal of ad spend concentration among fewer and fewer platforms and largely among big technology platforms, squeezing independent content developers and distributors. Over time, the new web economy model will evolve:
AI engine crawls content >> Users deploy AI agents for information>> AI Driven Results (AIDR) present content to users (no or highly reduced direct content/ website interaction)>> AI agent shows ads and earns revenue
This is not just a publisher problem. As users shift to AI agent managed interfaces, brands reputation and messaging are also at the mercy of AI agents.
It also creates a doom cycle for LLMs and AI agents. Less content on the open internet means less raw material for training and even less new content to generate useful information for their users, thus killing the value of their outputs. It is important for LLMs and AI agents to ensure that content development on the open and free internet continues to thrive.
Now is the right time to prepare for the imminent end of the old web economy and usher into the new web economy by building for content monetization and distribution in an AI driven world.
Today we are releasing an initial guidance and framework on how publishers can take control of their monetization and brands their content when integrating with AI agents. It will require publishers and brands to take control of how their content is accessed and used in a three step process:
- Create scarcity of content by managing and controlling access to bot traffic using robots.txt and web application firewall (WAF) management
- Develop LLM friendly content discovery by providing content access rules and content meta data in an LLM friendly mechanisms: Meta data, or LLMs.txt
- Monetize/ distribute content via cost per crawl (CPCr) APIs and / or LLM Ingest API. Both are new APIs introduced in this framework, including a variation for brands to use the LLM Ingest API without the pricing transactions.
In addition, we have reviewed how NLWeb protocol released by Microsoft can be used and integrated into the LLM ingest API.
These new tools could enable content owners to provide more relevant content to AI agents and also understand how and where their content is being used. The API proposals also allow for dynamically pricing content based on demand and monetizing in the new Agentic Web Economy.
As the digital advertising ecosystem evolves, we need to evolve with it. We’re eager to hear more from the industry and standardize the new web economy model. The time is now, to investigate, to lean in on new technology and new models for monetization. We can’t do this alone.
Learn more about IAB Tech Lab’s LLM Initiative here, and share your feedback by June 30, 2025.
Shailley Singh
COO and EVP, Product
IAB Tech Lab