CCBot
Common Crawl · AI / data crawler
Common Crawl's crawler, which builds Common Crawl's open web archive. Common Crawl publishes its IP ranges.
- User-agent contains
CCBot- robots.txt token
CCBot- Respects robots.txt
- Operator docs don't say / may not apply
- Official IP ranges
- 5 prefixes
How to verify CCBot
Matching the user agent isn't enough, because any client can send it. A real CCBot request comes from an IP address inside Common Crawl's published list:
- https://index.commoncrawl.org/ccbot.json5 prefixes · list dated 2026-08-11 · fetched 2026-10-21
The quickest way to check a whole log is our free log checker. It runs in your browser and splits CCBot requests into real and fake.
First 5 prefixes
2600:1f28:365:8000::/56 3.41.188.32/29 18.97.9.168/29 18.97.14.80/29 18.97.14.88/30
Block or allow CCBot in robots.txt
User-agent: CCBot Disallow: /
Use Allow: / instead to explicitly permit it, or build a full file with the AI robots.txt generator. robots.txt is a request, not a lock: it only works for bots that honor it, and only real CCBot traffic would.
Verify CCBot automatically
One API call tells your app, WAF or middleware whether a request claiming to be CCBot really comes from Common Crawl's official IP ranges.