What is CCBot's user-agent string?

CCBot/2.0 (https://commoncrawl.org/faq/) (vendor-documented at commoncrawl.org/ccbot)

CCBot

Q: What is CCBot?

CCBot is Common Crawl's training crawler — it crawls the web to collect content for model training. Blocking it opts your content out of future models but does not affect live answers.

Common Crawl · Training crawler · Updated 2026-07-19

CCBot is Common Crawl’s training crawler — it crawls the web to collect content for model training. Blocking it opts your content out of future models but does not affect live answers.

User-agent string

CCBot/2.0 (https://commoncrawl.org/faq/)

Source: vendor-documented at commoncrawl.org/ccbot.

Does CCBot respect robots.txt?

Respects robots.txt — Common Crawl documents the CCBot disallow as the way to prevent crawling, and warns that crawlers falsely identifying themselves as CCBot exist.

Allow or block CCBot

# Allow (recommended if you want CCBot traffic)
User-agent: CCBot
Allow: /

# Block
User-agent: CCBot
Disallow: /

Verifying it’s really CCBot

Forward-confirmed reverse DNS to *.crawl.commoncrawl.org, or the published ranges at index.commoncrawl.org/ccbot.json (Common Crawl recommends FCrDNS for IPv4; its IPv6 addresses have no reverse DNS yet). User-agent strings can be spoofed by anyone — identity claims are only trustworthy when the source IP matches the operator’s published ranges.

Fetch the published ranges and check a suspect IP against them:

curl -s https://index.commoncrawl.org/ccbot.json

What site owners should know

Common Crawl is a nonprofit open web archive, not an AI lab — but its dumps are the canonical source of LLM training data, which is why robots.txt surveys such as Cloudflare Radar consistently rank CCBot among the most fully-blocked crawlers. Blocking it keeps you out of future dumps only; already-published archives remain.

Scan your site to see how your robots.txt, content negotiation, and discovery files treat AI agents — including whether you’re blocking agents you meant to allow. Background: the robots.txt guide, how AI agents browse, and agent readiness.

All AI agent user-agents

ClaudeBot · Claude-User · Claude-SearchBot · GPTBot · OAI-SearchBot · ChatGPT-User · PerplexityBot · Perplexity-User · Google-Extended · Google-Agent · Google-CloudVertexBot · Meta-ExternalAgent · Meta-ExternalFetcher · MistralAI-User · Applebot · Applebot-Extended · Amazonbot · bingbot · DuckAssistBot · CCBot · Bytespider