Announcing “Content Monetization Protocols (CoMP) for AI” Working Group

ICYMI, IAB Tech Lab recently hosted a workshop on one of the most consequential impacts of AI on digital media in the open web – the significant drop in publisher traffic and monetization as a result of AI driven search and Agentic AI. This was a follow up to our recently announced LLM and AI Agent integration framework initiative aka LLM Content Ingest API to help publishers mitigate risk from impact of AI driven search and standardize framework for content monetization.

This post is a summary of highlights and conclusions from the workshop that will be driving IAB Tech Lab’s work in this area going forward.

I want to start with applause for all those who joined the workshop in person and virtually – 80 plus executives from publishers, edge cloud providers (Cloudflare, Fastly, AWS) and startups building content monetization solutions for publishers.

I especially appreciate Jonathan Roberts from Dotdash Meredith and Achim Schlosser (who flew in from Germany!) from Bertelsmann Group in advancing the publisher’s cause. Special thanks is also due to Shelley Susan from Meta for presenting the LLM perspectives. We were happy to have Toshit Panigrahi, Founder and CEO, Tollbit; Dappier Cofounders Krish Arvapally and Dan Goikhman; Arielle Kitei from Cloudflare; and Annelies Jansen and Joshua Freeman from ProRata.ai present underlying data on AI related bot traffic, its impact and different approaches and business models to mitigating risk to monetization.

The problem is real

There was wide consensus that this problem is real and is evolving fast, largely driven by a surge in AI bots due to proliferation of RAG applications, MCP servers and Agentic AI.

Attendees highlighted that the problem is exacerbated due to the following reasons:

It is hard for publishers as robots.txt, a widely relied upon method to instruct bots and scrapers about what they can and cannot do, is ignored a lot of the time.
It is very hard for publishers to get attention from the large technology companies. While there have been some content deals with large publishers, there is no voice for mid and long tail.
Scraping of content is only increasing at an exponential rate – with AI agent bots as new entrants that could grow into thousands to millions as more people start using AI based applications and consumer interfaces.
There is no marketplace or exchange for content like there is for advertising. So the question is what doors does a publisher knock and how?

Time for “a door with a lock” is now

Everyone in the room agreed that the time is now to implement measures to stop the widespread and uncontrolled scraping of publisher intellectual property. The consensus in the room was clear:

Develop a standard framework across infrastructures and enable publishers to exercise control and enforce their terms of use at scale.

Blocking and redirecting bots and scrapers is the only way to bring the problem to the attention of the AI ecosystem. The demand for IAB Tech Lab is to create the standard and buy in for adoption of enforcement mechanisms.

Build the foundations of a new marketplace?

Another aspect highlighted during the afternoon was the absence of a marketplace and methods to attribute contribution of content. The presenters from Tollbit, Cloudflare, Dappier and Prorata demonstrated that the demand exists and needs to be harnessed in a structured manner.

Though the business models are different, but there are some key foundational characteristics that can be standardized to develop a market:

Content discovery so LLMs and AI agents can understand and compare content easily for purchase
Content purpose and permission taxonomy
Outcomes or attribution of content usage or application

Taken together, this will create a market for discovery and purchase of content by LLMs and AI agents

Build the Easy button

Our speakers presented the different business models that could be used in great detail:

Pay per crawl
Aggregation or pay per use
Outcome or attribution based payment

It is clear that there will be different flavors of monetization that will coexist. The industry will need to support different methods from ‘all you can eat buffet’ (content library partnerships) to ‘pre-fixe menu’ (Pay per crawl) to more sophisticated ‘a la carte’ ( cost per query) options.

The attendees echoed the need for standard framework and currency definitions to make this approach work across providers and platforms.

Next Steps

Fortunately, the LLM and AI Agents Integration Framework already incorporates most of the suggestions in the three-step framework:

Block bot traffic i.e. door with a lock
LLM friendly discovery i.e. building a market
LLM Ingest API i.e. easy button for monetization

We are thankful to all the attendees for their vigorous participation in helping prioritize the work that needs to be done to support the industry.

Oh btw, there was one more piece of feedback- the originally announced name for this initiative, the LLM Content Ingest API, does not reflect the full breadth of the work that will be needed, so we are changing it.

We will be launching a new working group “Content Monetization Protocols (CoMP) for AI Working Group” to take this initiative further. Please fill out this form to register for the working group.