LLM Content Ingest API Initiative

The rapid rise of AI agents, large language models (LLMs), and AI-driven search summaries (e.g., Google’s Gemini-powered AI Overviews) is transforming how users access information. AI-driven search summaries are reducing publisher traffic by 20–60% (up to 90% for niche sites), with an estimated $2 billion ad revenue loss significantly impacting the economic model of the web. This shift, coupled with with 117% bot traffic surge (5.05M scrapes/site, 1.89M hidden scrapes) and 95.7% fewer referrals (0.37% vs. 8.63% Google) and the concentration of digital advertising revenue among a handful of tech giants (80.8% ad revenue is shared by the top 10 and 11.0% by the top 11–25), threatens the sustainability of publishers on the open internet. Without intervention, the content ecosystem that fuels AI systems risks collapse, degrading both publisher viability and AI quality. 

And it is not just the publishers that will be affected. As users shift to AI generated answers to their information needs, brand reputation and messaging will only be as good as the AI system the user is using. Thus it is necessary to ensure the AI systems have the right information about the brands and they are all able to get it from the source of truth- the brand itself in an organized manner.

This document is an initial recommended guidance for 

• Publishers and brands to control and provide guided access for LLM and AI agent bots
• Providing LLM friendly content access 
• Cost per crawl monetization opportunity
• Query API based monetization for content with variable pricing and bidding opportunities for fast moving or in demand content
• AI native technologies like Model Context Protocol (MCP) based NLWeb API

Learn more about the AI challenges for the web economy, actions publishers and brands can take and draft versions of recommended methods including suggested APIs for content monetization in our LLMs and AI Agents Integration framework.