Block AI training crawlers without losing search traffic
Most sites that decide to "block AI" end up blocking the crawlers that were sending them readers. The two behaviours travel under similar names and arrive from the same companies, but they are not the same thing — and robots.txt treats them separately if you name them separately.
Two different jobs, two different crawlers
Every major AI company runs at least two crawlers with different purposes:
- Training crawlers collect text to train future models. They send you nothing back. Blocking them costs you no traffic.
- Search and citation crawlers fetch pages so an assistant can answer a question and link to you. Blocking them removes you from AI answers — this is where referral traffic disappears.
The common mistake: adding User-agent: * with Disallow: / for "AI", or copying a blocklist that lumps both categories together. You keep the training exposure you were worried about roughly the same, and lose the citations that were bringing visitors.
Which crawler is which
| User agent | Operator | What it does | Blocking it costs you |
|---|---|---|---|
| GPTBot | OpenAI | Training data collection | Nothing in traffic terms |
| OAI-SearchBot | OpenAI | Indexing for ChatGPT search results | Citations and referral clicks |
| ChatGPT-User | OpenAI | Live fetch when a user asks about your page | Your page in that user's answer |
| ClaudeBot | Anthropic | Training data collection | Nothing in traffic terms |
| Claude-SearchBot | Anthropic | Indexing for Claude search | Citations and referral clicks |
| Claude-User | Anthropic | Live fetch on user request | Your page in that user's answer |
| Google-Extended | Gemini training opt-out token | Nothing — it is not a crawler, it is a permission flag | |
| Googlebot | Web search index (also feeds AI Overviews) | Your entire Google presence | |
| PerplexityBot | Perplexity | Indexing for answers with citations | Citations and referral clicks |
| CCBot | Common Crawl | Open dataset used by many trainers | Nothing in traffic terms |
| Bytespider | ByteDance | Training data collection | Nothing in traffic terms |
| Amazonbot | Amazon | Alexa answers and training | Limited |
Google-Extended deserves a note because it confuses people: it is not a separate crawler that visits your site. Googlebot does the crawling; Google-Extended is the token you disallow to opt out of Gemini training while keeping normal search indexing. Disallowing it does not remove you from Google Search.
The configuration that keeps your traffic
This is the search-only posture: deny training, allow citation.
# Training crawlers — denied User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Bytespider User-agent: Google-Extended Disallow: / # Search and citation crawlers — allowed User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot Allow: / Disallow: /admin Disallow: /api # Everyone else User-agent: * Allow: / Disallow: /admin Disallow: /api Sitemap: https://example.com/sitemap.xml
The rule that breaks most robots.txt files: a crawler that matches a named User-agent group ignores the * group completely. It does not merge them. So every named group needs its own full set of Disallow lines — if you put Disallow: /admin only under *, the named crawlers you just allowed will happily crawl /admin.
robots.txt is a request, not a control
robots.txt is voluntary. Well-behaved crawlers honour it; scrapers that spoof user agents do not. If the content genuinely must not be taken, you need enforcement at the request layer:
- Verify, do not trust the user agent. Major operators publish IP ranges or support reverse DNS verification. A request claiming to be GPTBot from an unlisted address is just an anonymous request.
- Rate limit by behaviour. Sustained sequential crawling of every URL looks nothing like human browsing, whatever the user agent says.
- Put real value behind authentication. Anything reachable without a session will eventually be collected by someone.
This is where a policy layer earns its place: robots.txt states intent, and the request boundary enforces it for clients that ignore the statement.
Choosing a posture
- open — content marketing, documentation, anything where being cited is the point. Maximise AI visibility.
- search-only — the sensible default for most commercial sites. Keep the referral traffic, decline to be training data.
- verified-only — original research, paid content, proprietary data. Named crawlers from verifiable addresses pass; spoofed agents get ordinary traffic rules.
- closed — internal tools, customer dashboards, anything that should not appear in an answer at all.
Whichever you pick, write it down as configuration rather than a one-off edit, and keep the reasoning next to it. The crawler landscape changes every few months; a policy you can diff and re-run is worth more than a robots.txt someone hand-edited last year.
Verify it actually works
Do not trust the file — fetch your own site as each crawler and read the status code:
# Should be allowed (search/citation) curl -sI -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0)" https://example.com/ | head -1 curl -sI -A "Mozilla/5.0 (compatible; PerplexityBot/1.0)" https://example.com/ | head -1 # Should be denied (training) curl -sI -A "Mozilla/5.0 (compatible; GPTBot/1.1)" https://example.com/ | head -1 # And read the served file itself curl -s https://example.com/robots.txt
One more thing worth checking if you are on Cloudflare: the dashboard has a managed robots.txt / AI crawler control that can override the file you deployed. A policy that looks correct in your repository and behaves differently in production is usually this setting.
How TJ Sentinel handles this
TJ Sentinel treats the AI posture as one field in a policy file. You choose open, search-only, verified-only or closed, and it generates the matching robots.txt, llms.txt and request-layer rules for your stack — with the named-group trap handled, and the same verdict whether the request hits a static host, Express, Next.js or a Cloudflare Worker. Blocked requests carry a stable reason code so you can tell a policy decision from a bug.