The complete robots.txt list for AI crawlers

Reference table of the named AI crawlers you are likely to see in access logs: who runs each one, what it is collecting, whether it honours robots.txt, and what you give up by denying it.

The crawlers

Grouped by what they actually do, because that is the decision you are making — not which company sent them.

Training data collection

These fetch pages to build training corpora. They send no visitors back. Denying them has no traffic cost.

Search indexing and citation

These decide whether an AI assistant can surface and link to your page. Denying them removes you from AI answers — this is where referral traffic is lost.

Directives that actually work

Named groups do not inherit from the wildcard group

This is the single most common robots.txt bug. A crawler matching a named User-agent line reads only that group and ignores User-agent: * completely. The groups are not merged.

# BROKEN — PerplexityBot can now crawl /admin
User-agent: *
Disallow: /admin

User-agent: PerplexityBot
Allow: /

# Every named group repeats its own Disallow lines
User-agent: *
Disallow: /admin

User-agent: PerplexityBot
Allow: /
Disallow: /admin

Multiple user agents can share one group

Consecutive User-agent lines with no directives between them form a single group, which keeps the file readable:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

Crawl-delay is not part of the standard

Google ignores Crawl-delay entirely. Some crawlers honour it, most do not. If you need to control request rate, do it at the request layer with real rate limiting — a directive that is ignored is not a control.

Verifying a crawler is who it claims to be

User agents are trivially spoofed. A request claiming to be GPTBot proves nothing. The operators that publish verification methods make this checkable:

• Published IP ranges. OpenAI, Anthropic and Perplexity publish JSON files listing the addresses their crawlers use. Fetch, cache and compare.
• Reverse DNS then forward confirm. Google's long-standing method: reverse-lookup the IP, check the hostname is under the expected domain, then forward-resolve that hostname and confirm it matches the original IP. One direction alone is forgeable.
• Behaviour. Sequential enumeration of every URL, no asset requests, no referrer, perfectly regular intervals — that is a crawler regardless of what the header says.

Check what you are actually serving

# The file as served (not as committed)
curl -s https://example.com/robots.txt

# Behaviour per crawler identity
for ua in "GPTBot/1.1" "OAI-SearchBot/1.0" "PerplexityBot/1.0"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" -A "Mozilla/5.0 (compatible; $ua)" https://example.com/)
  echo "$ua -> $code"
done

Two deployment-level surprises worth ruling out when the served file does not match your repository: a host-level managed robots.txt feature overriding yours (Cloudflare has one under AI Crawl Control, and it is on by default for some plans), and a CDN serving a cached copy from before your last deploy.

Generating this from policy

Maintaining these groups by hand goes stale — the crawler list changes every few months, and the named-group trap resurfaces every time someone adds a rule. TJ Sentinel takes an AI posture (open, search-only, verified-only, closed) and emits the matching robots.txt with every named group complete, plus the request-layer rules that enforce it for clients which ignore the file.

Generate a policy for your site → · Read: keep search traffic while denying training →