IAB Tech Lab Spiders and Bots List

In addition to the primary purpose of avoiding billing customers, and paying suppliers for nonhuman traffic, using an industry standard list like Spiders and Robots list reduces discrepancies between parties and achieves the requirement for detecting General Invalid Traffic as per the Invalid Traffic Detection and Filtration Guidelines. This benefits all stakeholders in the supply chain.

The list provides the industry two main purposes.  First, the spiders and robots list consists of two text files: one for valid browsers or user agents and one for known robots. These lists are intended to be used together to comply with the “dual pass” approach to filtering as defined in the IAB’s Ad Impression Measurement Guidelines (i.e., identify valid transactions using the valid browser list and then filter/remove invalid transactions using the known robots list).  Second, the spiders and robots list supports the MRC’s General Invalid Traffic Detection and Filtration Standard by providing a common industry resource and list for facilitating IVT detection and filtration.