What is web-bot-auth?
Web-bot-auth is an emerging standard for AI agents to prove their identity using cryptographic HTTP message signatures (RFC 9421). Instead of relying on User-Agent strings — which can be faked — agents sign their requests with a private key. Sites verify the signature against a published public key.
Why it matters
User-Agent detection is no longer reliable:
- ChatGPT Agent Mode uses a fake Chrome UA (
Chrome/138.0.0.0) — indistinguishable from a browser by UA alone - Perplexity runs stealth crawlers (3-6M requests/day) with generic Chrome UAs
- Google-Agent uses standard Chrome UA strings
The industry is moving to cryptographic identity. OpenAI, Google, Cloudflare, Shopify, Vercel, and Visa are backing this approach. IETF standardization is in progress.
What AgentGrade checks
Signatures directory published — We check for /.well-known/http-message-signatures-directory. This is the standard location where a site publishes its agent identity and the public keys that can be used to verify its signed requests.
Members declared — The directory should contain a members array listing the agent identities (e.g., "ChatGPT", "GoogleAgent") that the site recognizes or acts as.
Public keys available — Each member should have a publicKeyUrl pointing to a fetchable public key for signature verification.
How to implement
Publish a JSON file at /.well-known/http-message-signatures-directory:
{
"members": [
{
"name": "my-agent",
"publicKeyUrl": "https://example.com/.well-known/keys/agent.pub"
}
]
}
To verify incoming signed requests from known agents (like ChatGPT), use the web-bot-auth npm package:
npm install web-bot-auth
Known agent directories
| Agent | Directory URL |
|-------|---------------|
| ChatGPT | https://chatgpt.com/.well-known/http-message-signatures-directory |