Guide · nginx

Block fake AI bots in nginx

Anyone can send User-Agent: GPTBot. In our 15-day log study at least 43% of requests claiming to be AI crawlers were vulnerability scanners. This config lets real AI bots through and returns 403 to the impostors, by checking each request's IP against the list the bot's operator publishes.

17 bots, 3,021 IP ranges, built 2026-10-21 from the official lists. Free, no sign-up.

1. Install

Save the file next to your nginx config and include it inside the http block:

sudo curl -o /etc/nginx/conf.d/realbots.conf \
  https://aibots.kaderagha.com/data/v1/nginx-fake-ai-bots.conf

Files in conf.d/ are included in http on most distributions; otherwise add include /path/to/realbots.conf; there. Then, in every server block you want to protect:

if ($realbots_fake) { return 403; }

Test and reload:

sudo nginx -t && sudo systemctl reload nginx

2. Check it works

From your own computer (which is not an OpenAI server), pretend to be GPTBot. You should get 403, and a normal request should still get 200:

curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot/1.2" https://example.com/   # 403
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/                      # 200

3. Behind Cloudflare or a load balancer?

Then nginx sees the proxy's IP, not the visitor's, and every bot would look fake. Set up ngx_http_realip_module first so $remote_addr is the real client. For Cloudflare that means a set_real_ip_from line for each of Cloudflare's IP ranges plus real_ip_header CF-Connecting-IP;. Only trust that header from the proxy's own ranges: otherwise anyone can forge it. Not sure? Upload a log to the checker: it tells you when your log shows proxy IPs instead of visitors.

How it works

1. A map on $http_user_agent finds which bot the request claims to be (Googlebot is matched last, since other Google crawlers contain that token too).
2. A geo block maps the client IP to the bots allowed to use it, from the operators' published IP lists.
3. A second map sets $realbots_fake to 1 when a bot is claimed and the IP isn't on its list. Requests that don't claim a bot are never touched.

Google-Extended is included on purpose: Google documents it as a robots.txt token with no HTTP user agent of its own, so a request that sends it is never Google.

Bots it checks

Bots whose operators publish no IP list (for example Bytespider and Meta's crawlers) can't be verified, so the config leaves them alone. To refuse those, use robots.txt.

• GPTBot (OpenAI)
• OAI-SearchBot (OpenAI)
• ChatGPT-User (OpenAI)
• OAI-AdsBot (OpenAI)
• ClaudeBot (Anthropic)
• Claude-SearchBot (Anthropic)
• Claude-User (Anthropic)
• PerplexityBot (Perplexity)
• Perplexity-User (Perplexity)
• Google-Extended (Google)
• Bingbot (Microsoft)
• Applebot (Apple)
• CCBot (Common Crawl)
• DuckAssistBot (DuckDuckGo)
• MistralAI-User (Mistral AI)
• Google-CloudVertexBot (Google)
• Googlebot (Google)

Keep it current

Operators add IP ranges without notice, and a real bot on a new range would get 403 until you update. We rebuild the file every night from the official lists. A weekly cron is enough:

0 4 * * 1  curl -fsS -o /etc/nginx/conf.d/realbots.conf https://aibots.kaderagha.com/data/v1/nginx-fake-ai-bots.conf && nginx -t && systemctl reload nginx

Prefer raw data? The same lists are in the JSON feed /data/v1/ranges.json. On WordPress, the Verified AI Bot Analytics plugin does the same check without touching nginx.