Research · 2026-10 · updated monthly

Which AI crawlers do the top 1,000 websites block?

We read the robots.txt of the world's 1,000 most popular domains and checked, for 22 AI and search crawlers, whether each site shuts it out. About one in four sites blocks AI training, but most keep the door open to AI search.

Share of sites that block each crawler

Base: 591 domains with a reachable website (517 serve a robots.txt; the rest have none and block nothing).

What stands out

• Training is blocked far more than search. 18.1% block GPTBot (OpenAI training) but only 10.8% block OAI-SearchBot (ChatGPT search). 7.3% of sites block GPTBot while still letting ChatGPT search in, for example www.linkedin.com, whatsapp.net, whatsapp.com, yahoo.com, snapchat.com.
• The big social platforms block every training crawler while staying open to Google Search: facebook.com, instagram.com, twitter.com, www.linkedin.com, x.com, pinterest.com.
• Only 27.7% of sites mention any AI bot by name. Most sites don't have an AI policy in robots.txt at all; the rest of the blocking comes from sites that disallow everything.
• Newer bots are rarely named: Amzn-SearchBot (2.2%), OAI-AdsBot (0.7%), MistralAI-User (6.6%). Where they're blocked, it's usually by a blanket rule.

Methodology

• Sites: Tranco 467YX (2026-10-18), top 1,000 domains. Fetched 2026-10-19 from https://<domain>/robots.txt, falling back to www.
• Parsing follows RFC 9309: a crawler uses the group naming its token, otherwise the * group; longest matching rule wins, Allow wins ties. "Blocked" means both / and a typical page path are disallowed. Partial blocks (some folders) don't count.
• 409 domains had no reachable website (CDN/API/infrastructure domains) or refused the request, and are excluded.
• robots.txt is a request, not enforcement: it says what a site asks for, not what crawlers do.