AI crawler directory
User-agent tokens, purposes and official IP ranges for the AI and search bots that visit websites. Purposes are taken from each operator's own documentation; IP lists are downloaded from the operator daily.
| Bot | Operator | Type | Official IP ranges |
|---|---|---|---|
| GPTBot | OpenAI | Model training crawler | 18 prefixes |
| OAI-SearchBot | OpenAI | AI search indexer | 39 prefixes |
| ChatGPT-User | OpenAI | User-triggered fetcher | 234 prefixes |
| OAI-AdsBot | OpenAI | Ads validation | 2 prefixes |
| ClaudeBot | Anthropic | Model training crawler | 14 prefixes |
| Claude-SearchBot | Anthropic | AI search indexer | 14 prefixes |
| Claude-User | Anthropic | User-triggered fetcher | 14 prefixes |
| PerplexityBot | Perplexity | AI search indexer | 8 prefixes |
| Perplexity-User | Perplexity | User-triggered fetcher | 4 prefixes |
| Google-Extended | robots.txt token only | no user agent | |
| Googlebot | AI search indexer | 2,161 prefixes | |
| Bingbot | Microsoft | AI search indexer | 28 prefixes |
| Applebot | Apple | AI search indexer | 24 prefixes |
| CCBot | Common Crawl | AI / data crawler | 5 prefixes |
| DuckAssistBot | DuckDuckGo | User-triggered fetcher | 486 prefixes |
| MistralAI-User | Mistral AI | User-triggered fetcher | 4 prefixes |
| Bytespider | ByteDance | AI / data crawler | none published |
| meta-externalagent | Meta | Model training crawler | none published |
| meta-externalfetcher | Meta | User-triggered fetcher | none published |
| Amazonbot | Amazon | Model training crawler | none published |
| Amzn-SearchBot | Amazon | AI search indexer | none published |
| Google-CloudVertexBot | AI / data crawler | 2,161 prefixes |
Lists: 17 official JSON files from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Common Crawl, DuckDuckGo and Mistral.