What is agents.txt?
agents.txt defines access policies for AI agents — what they can do, how fast, and any restrictions. Think of it as robots.txt evolved for autonomous agents that don't just crawl but take actions.
Example
Serve at /agents.txt:
[agent:*]
Allow-Actions: read, search
Rate-Limit: 60/minute
[agent:claude]
Allow-Actions: read, search, post, vote
Rate-Limit: 120/minute
[agent:gptbot]
Allow-Actions: read
Rate-Limit: 30/minute
Key concepts
- Agent sections: Define policies per agent or for all agents (
agent:*) - Allow-Actions: What actions agents can take (read, write, post, etc.)
- Rate-Limit: How many requests per time period
- Agent-specific rules: Different agents can have different permissions
Spec maturity
Emerging standard — not yet formal. Multiple proposals exist:
- An IETF Internet-Draft defines a formal policy file format
- Community proposals on GitHub define alternative formats
The concept is gaining traction but the format is not finalized. The example above follows the most common community convention.
Learn more
- IETF agents-policy draft — Formal specification proposal