AI robots.txt generator
Decide which AI bots may read your site. Training crawlers, AI search indexers and user-triggered fetchers are separate bots with separate robots.txt tokens, so you can opt out of training without disappearing from AI answers.
Model training crawler
- GPTBotOpenAI
- ClaudeBotAnthropic
- meta-externalagentMeta
- AmazonbotAmazon
AI / data crawler
- CCBotCommon Crawl
- BytespiderByteDance
- Google-CloudVertexBotGoogle
robots.txt token only
- Google-ExtendedGoogle
AI search indexer
- OAI-SearchBotOpenAI
- Claude-SearchBotAnthropic
- PerplexityBotPerplexity
- ApplebotApple
- Amzn-SearchBotAmazon
User-triggered fetcher
- ChatGPT-UserOpenAI
- Claude-UserAnthropic
- Perplexity-UserPerplexity
- DuckAssistBotDuckDuckGo
- MistralAI-UserMistral AI
- meta-externalfetcherMeta
Ads validation
- OAI-AdsBotOpenAI
Your robots.txt
# AI crawler rules — generated at aibots.kaderagha.com/tools/robots-txt-generator # Blocked AI bots User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot User-agent: Bytespider User-agent: meta-externalagent User-agent: Amazonbot User-agent: Google-CloudVertexBot Disallow: / # Allowed AI bots User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: OAI-AdsBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Applebot User-agent: DuckAssistBot User-agent: MistralAI-User User-agent: meta-externalfetcher User-agent: Amzn-SearchBot Allow: / # Everyone else (incl. Googlebot, Bingbot) User-agent: * Allow: /
Put it at https://yoursite.com/robots.txt. robots.txt is a request: real crawlers from these operators say they honor it, but fakes ignore it. Check your log to see who actually visits.