What is Bytespider's user-agent string?

Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/) (community-documented — ByteDance publishes no reachable official documentation; the UA’s own reference link (zhanzhang.toutiao.com) is inaccessible outside China)

Bytespider

Q: What is Bytespider?

Bytespider is ByteDance's training crawler — it crawls the web to collect content for model training. Blocking it opts your content out of future models but does not affect live answers.

ByteDance · Training crawler · Updated 2026-07-19

Bytespider is ByteDance’s training crawler — it crawls the web to collect content for model training. Blocking it opts your content out of future models but does not affect live answers.

User-agent string

Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)

Source: community-documented — ByteDance publishes no reachable official documentation; the UA’s own reference link (zhanzhang.toutiao.com) is inaccessible outside China.

Does Bytespider respect robots.txt?

No operator statement exists. Multiple independent server-log analyses report Bytespider fetching robots.txt but crawling disallowed URLs anyway — treat compliance as unverified.

Allow or block Bytespider

# Allow (recommended if you want Bytespider traffic)
User-agent: Bytespider
Allow: /

# Block
User-agent: Bytespider
Disallow: /

Verifying it’s really Bytespider

Nothing published — no IP list and no reverse-DNS pattern. Sites that need a firm block typically filter by ASN or behavior. User-agent strings can be spoofed by anyone — identity claims are only trustworthy when the source IP matches the operator’s published ranges.

What site owners should know

ByteDance’s crawler (feeding products including the Doubao models) ranks among the highest-volume AI crawlers measured by Cloudflare Radar, yet has no official documentation, IP list, or robots.txt statement — the biggest transparency gap in this list.

Scan your site to see how your robots.txt, content negotiation, and discovery files treat AI agents — including whether you’re blocking agents you meant to allow. Background: the robots.txt guide, how AI agents browse, and agent readiness.

All AI agent user-agents

ClaudeBot · Claude-User · Claude-SearchBot · GPTBot · OAI-SearchBot · ChatGPT-User · PerplexityBot · Perplexity-User · Google-Extended · Google-Agent · Google-CloudVertexBot · Meta-ExternalAgent · Meta-ExternalFetcher · MistralAI-User · Applebot · Applebot-Extended · Amazonbot · bingbot · DuckAssistBot · CCBot · Bytespider