RFC 9309 — Robots Exclusion Protocol

Discovery · IETF · Updated 2026-07-19

Status: Proposed Standard (IETF, 2022) — the formalization of 30 years of robots.txt practice · Authoritative source

The rules of engagement for crawlers, now doing a job it was never designed for: expressing per-vendor AI policy across training, search, and user-fetch tiers through User-agent tokens.

RFC 9309 standardizes what robots.txt always was informally: groups addressed to User-agent tokens, Allow/Disallow rules, most-specific-match wins, and — crucially — voluntary compliance. Nothing in the protocol enforces anything.

The agent era stretched it in one specific way: every AI vendor now runs several bots with separate tokens and separate compliance behavior (training crawlers honor it; search crawlers honor it; user-initiated fetchers mostly document that they may not). A robots.txt that treats “AI” as one thing misses the actual control surface. The robots.txt guide covers the tier model; the user-agent reference documents every token.

The AI-policy pattern

# Block training, stay citable
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Search-index and user-fetch tiers stay allowed by default

How AgentGrade checks it

The scanner parses your robots.txt per-token, checks whether documented AI agents are allowed or blocked (including the policy tokens that never appear in logs), and flags blocks that contradict your apparent intent. Build a policy interactively with the generator. Scan your site to see where you stand.

Related

• robots.txt for agents — the guide
• robots.txt Generator
• Every documented AI crawler token

All standards

RFC 9421 · Web Bot Auth · RFC 8615 · MCP · A2A · x402 · WebMCP · llms.txt · RFC 9309 · RFC 9110 · Index