Guide · Apache

Block fake AI bots with .htaccess

Blocking “GPTBot” by user agent also blocks the real GPTBot, and lets every impostor that picks a different name through. These Apache rules do it the other way round: a request that claims to be an AI crawler gets 403 only if its IP isn't on the list the bot's operator publishes. In our log study at least 43% of AI-bot requests were such impostors.

Get the rules (.txt)Using nginx?On Cloudflare?

17 bots, 3,021 IP ranges, built 2026-10-21 from the official lists. Free, no sign-up.

1. Install

Needs Apache 2.4 with mod_rewrite (on by default at most hosts). Best: paste the file inside your <VirtualHost> and reload Apache. On shared hosting, paste it at the top of the .htaccess in your site root, above any WordPress block.

curl -o realbots.txt https://aibots.kaderagha.com/data/v1/htaccess-fake-ai-bots.txt
# vhost: Include /path/to/realbots.txt   (then: sudo apachectl configtest && sudo systemctl reload apache2)

Speed: Apache re-reads .htaccess on every request. The file has a few thousand lines, and in our test on Apache 2.4 it added about 18 ms per request. In the vhost config it is read once at startup.

2. Check it works

From your own computer, pretend to be GPTBot (403), then make a normal request (200):

curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot/1.2" https://example.com/   # 403
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/                      # 200

3. Behind Cloudflare or a proxy?

Apache then sees the proxy's address and every bot looks fake. Enable mod_remoteip with RemoteIPHeader CF-Connecting-IP and a RemoteIPTrustedProxy line for each of Cloudflare's IP ranges, so the header is only trusted from Cloudflare. Our log checker tells you if your log shows proxy IPs instead of visitors.

How the rules work

Each bot gets one block like this (shortened):

# GPTBot (OpenAI)
RewriteCond %{HTTP_USER_AGENT} GPTBot [NC]
RewriteCond expr "! -R '132.196.86.0/24'"
RewriteCond expr "! -R '...next official range...'"
RewriteRule ^ - [F]

Conditions are ANDed: the request is refused only if it says GPTBot and its IP is outside every OpenAI GPTBot range. Google-Extended has no ranges at all: Google documents it as a robots.txt token with no HTTP user agent, so a request carrying it is never Google. Bots whose operators publish no IP list (Bytespider, Meta, Amazonbot) aren't checked; refuse those with robots.txt.

Bots it checks

GPTBot (OpenAI)
OAI-SearchBot (OpenAI)
ChatGPT-User (OpenAI)
OAI-AdsBot (OpenAI)
ClaudeBot (Anthropic)
Claude-SearchBot (Anthropic)
Claude-User (Anthropic)
PerplexityBot (Perplexity)
Perplexity-User (Perplexity)
Google-Extended (Google)
Bingbot (Microsoft)
Applebot (Apple)
CCBot (Common Crawl)
DuckAssistBot (DuckDuckGo)
MistralAI-User (Mistral AI)
Google-CloudVertexBot (Google)
Googlebot (Google)

Keep it current

Operators add IP ranges without notice; until you update, a real bot on a new range gets 403. We rebuild the file nightly from the official lists, so re-download it every week or two. On WordPress, the Verified AI Bot Analytics plugin does the same check and refreshes its IP lists daily.

Not on this stack, or building your own bot filter?

Call the verification API from any language: send the IP and user agent, get verified / fake back. Same official IP lists, refreshed daily.

Get a free API key10,000 checks/month free · Pro $19/month for 1M