The IAB Spiders and Robots list helps companies identify automated traffic such as search engine crawlers, monitoring tools, and other nonhuman traffic that they don’t want showing up in their analytics and billable counts. Additionally, using an industry standard list like Spiders and Robots list reduces discrepancies between parties and achieves the requirement for detecting General Invalid Traffic as per the Invalid Traffic Detection and Filtration Guidelines.
Though members get discounts, you do not have to be a member of the IAB to subscribe to the list. To subscribe to the list please follow the “How to Access” steps outlined below. Subscriptions last 12 months from the date of sign up and access is provided via FTP. Access to the list is not available until payment and a signed data subscription agreement have been received.
Email spiders@iab.com and request to be invoiced by the Spiders and Bots team. Only one account login is granted access per subscribing company. When emailing to request an invoice please be sure to provide:
Once you receive an invoice from the Spiders and Bots team, the invoice must be paid in full along with the data subscription agreement signed and returned before FTP access to download the list is received. Access will be revoked if either is not complete within the invoice terms.
A subscription lasts for a year and gives you access to:
Pricing
IAB requests that subscribers submit any new browsers or spiders and/or robots they discover as often as possible. List modifications may be recommended by any entity including non-IAB members. The list will be updated monthly to reflect changes that are brought to the attention of The Alliance for Audited Media, ABC UK, and the Policy Board.
The IAB has enlisted The Alliance for Audited Media to perform evaluation and management services associated with maintaining and publishing the industry Spiders and Robots list, the use of which is required for compliance to the IAB’s Client Side Counting (CSC) Measurement Guidelines for “double pass” filtering. We have also formed a Spiders & Robots Policy Board to oversee and approve list modifications (see list below).