How do I block AI bots with robots.txt?

Add a group per bot token: "User-agent: GPTBot" followed by "Disallow: /". Each AI bot has its own token, and blocking one does not affect the others — vendors run separate crawlers for training, search indexing, and live user fetches. This generator writes the groups for every documented bot.

Can I block AI training but stay visible in AI search?

Yes — that is the most common middle position. Block the training tokens (GPTBot, ClaudeBot, CCBot, meta-externalagent, Google-Extended, Applebot-Extended) and leave search-index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed. The "Block training, stay citable" preset writes exactly this.

Do all AI bots obey robots.txt?

No. Training and search crawlers from the major vendors document that they obey it. User-initiated fetchers mostly document that they may not, because a human requested the page. robots.txt is a published preference — firm blocking requires filtering by the vendors’ published IP ranges.

robots.txt generator for AI crawlers

Updated 2026-07-19 · 21 documented bots

Pick a rule per bot — or start from a preset — and copy the generated robots.txt block. Every token comes from the vendor-documented user-agent reference; the tier column tells you what each rule actually does (blocking a training crawler does not affect search citations or live user fetches — the robots.txt guide explains why).

Bot	Operator	Tier	robots.txt behavior
ClaudeBot	Anthropic	Training crawler	Honors robots.txt
Claude-User	Anthropic	User-initiated fetcher	Honors robots.txt
Claude-SearchBot	Anthropic	Search-index crawler	Honors robots.txt
GPTBot	OpenAI	Training crawler	Honors robots.txt
OAI-SearchBot	OpenAI	Search-index crawler	Honors robots.txt
ChatGPT-User	OpenAI	User-initiated fetcher	May ignore robots.txt
PerplexityBot	Perplexity	Search-index crawler	Honors robots.txt
Perplexity-User	Perplexity	User-initiated fetcher	May ignore robots.txt
Google-Extended	Google	Training crawler	Policy token — no crawler of its own
Google-Agent	Google	User-initiated fetcher	May ignore robots.txt
Google-CloudVertexBot	Google	Owner-commissioned crawler	Honors robots.txt
Meta-ExternalAgent	Meta	Training crawler	Honors robots.txt
Meta-ExternalFetcher	Meta	User-initiated fetcher	May ignore robots.txt
MistralAI-User	Mistral AI	User-initiated fetcher	Honors robots.txt
Applebot	Apple	Search-index crawler	Honors robots.txt
Applebot-Extended	Apple	Training crawler	Policy token — no crawler of its own
Amazonbot	Amazon	Training crawler	Honors robots.txt
bingbot	Microsoft	Search-index crawler	Honors robots.txt
DuckAssistBot	DuckDuckGo	Search-index crawler	Honors robots.txt
CCBot	Common Crawl	Training crawler	Honors robots.txt
Bytespider	ByteDance	Training crawler	Unverified — compliance contested

Sitemap URL (recommended):

Your robots.txt

# Pick rules above — output appears here

Copied

Already have a robots.txt? Scan your site to see how it treats every documented agent — including bots you meant to allow but blocked.

Notes

User-initiated fetchers (ChatGPT-User, Perplexity-User, Google-Agent, Meta-ExternalFetcher) are listed, but most vendors document that this tier may ignore robots.txt — a rule for them is a preference, not a control. Firm blocks need IP filtering; verification endpoints are on each bot page.
Google-Extended and Applebot-Extended never send requests — they are policy tokens read by Googlebot and Applebot. "Allow" rules for them are omitted from the output (the default is already allowed); "Block" writes the opt-out group.
Bytespider has no vendor-documented robots.txt compliance; a block rule is included if selected, but treat it as advisory.