IAB/ABC International Spiders and Bots List

Last updated: July 27, 2020

What is it?

The IAB Spiders and Robots list helps companies identify automated traffic such as search engine crawlers, monitoring tools, and other nonhuman traffic that they don’t want showing up in their analytics and billable counts. Additionally, using an industry standard list like Spiders and Robots list reduces discrepancies between parties and achieves the requirement for detecting General Invalid Traffic as per the Invalid Traffic Detection and Filtration Guidelines.

How to Subscribe

Though members get discounts, you do not have to be a member of the IAB to subscribe to the list. To subscribe to the list please follow the “How to Access” steps outlined below.  Subscriptions last 12 months from the date of sign up and access is provided via FTP. Access to the list is not available until payment and a signed data subscription agreement have been received.

How to Access

Email spiders@iab.com and request to be invoiced by the Spiders and Bots team. Only one account login is granted access per subscribing company. When emailing to request an invoice please be sure to provide:

  1. A signed copy of the Spiders and Bots data subscription agreement. Please note that we do not accept redlines to these agreements as the terms are not negotiable.
  2. Your Accounts Payable contact information.
  3. An email address for the primary contact at your company who will receive all communications regarding your Spiders & Bots subscription. We recommend that you use an email alias if a team will be accessing the list.
  4. If your company requires a PO for purchase, please request that as well.

Once you receive an invoice from the Spiders and Bots team, the invoice must be paid in full along with the data subscription agreement signed and returned before FTP access to download the list is received. Access will be revoked if either is not complete within the invoice terms.

Subscription & Pricing

A subscription lasts for a year and gives you access to:

  • Industry leaders in spiders & bots detection and filtering
  • The Alliance for Audited Media and IAB support
  • A comprehensive guide to best practices in using the spiders & bots list

Pricing

  • IAB US General Member: $5,000
  • IAB Tech Lab Member: $5,000
  • IAB Associate Member: $7,500
  • Non-Member: $15,000

Submit Newly Discovered Browsers or Spiders & Bots

IAB requests that subscribers submit any new browsers or spiders and/or robots they discover as often as possible. List modifications may be recommended by any entity including non-IAB members. The list will be updated monthly to reflect changes that are brought to the attention of The Alliance for Audited Media, ABC UK, and the Policy Board.

About the International IAB/ABC Spiders & Bots List

The IAB has enlisted The Alliance for Audited Media to perform evaluation and management services associated with maintaining and publishing the industry Spiders and Robots list, the use of which is required for compliance to the IAB’s Client Side Counting (CSC) Measurement Guidelines for “double pass” filtering. We have also formed a Spiders & Robots Policy Board to oversee and approve list modifications (see list below).

Spiders & Bots Policy Board

  • Richard Gibbons, ABC UK
  • Martin Liljenback, Adobe
  • Steve Guenther, Alliance for Audited Media
  • Todd Martens, Alliance for Audited Media
  • Antoine Comets, Criteo
  • João Matos, Criteo
  • Thomas Kent, Extreme Reach
  • Matthew Katz, Freewheel
  • Breen Baker, Google
  • Zhong Wang, Google
  • Michael Smith, Hearst
  • Jaelene Price, Microsoft
  • Jill Wittkopp, IAB Tech Lab
  • Saif Jafri, Yahoo