Network reference
Web crawler IP ranges
Browse ranges used by general-purpose web crawlers and automated page fetchers outside the search, AI, advertising, and SEO categories.
Some operators publish one range family for several named agents. A matching address supports source verification, but a User-Agent name alone does not authenticate a bot.
Web crawlers
12 Web crawler ranges
| Bot | Type | IPv4 prefixes | IPv6 prefixes |
|---|---|---|---|
| Amazonbot | Web crawler | 1292 | 0 |
| Arquivo Web Crawler | Web crawler | 1 | 1 |
| CCBot | Web crawler | 4 | 1 |
| HubSpot Crawler HubSpot Feed Fetcher, HubSpot Page Fetcher | Web crawler | 36 | 0 |
| IndeedJobBot | Web crawler | 3 | 0 |
| New York Times Newsgathering | Web crawler | 6 | 0 |
| Parse.ly crawler Parse.ly | Web crawler | 10 | 0 |
| Prerender | Web crawler | 9 | 0 |
| SentiBot | Web crawler | 20 | 0 |
| ServerHunterSpider | Web crawler | 3 | 4 |
| Skroutz ImageBot SkroutzBot | Web crawler | 7 | 1 |
| WP Compress Optimizer WP-Compress/SaaS | Web crawler | 7 | 2 |