## How agentgrade scans your site

agentgrade is a diagnostic tool — like SSL Labs for agent-readiness. When someone enters a domain, we send a small set of targeted HTTP requests to check what agent-facing capabilities the site exposes.

### What we check

- **~15-20 requests** to well-known paths (`/.well-known/x402.json`, `/openapi.json`, `/robots.txt`, etc.)
- **HTTP headers** on responses (Payment-Required, WWW-Authenticate, Content-Type)
- **Machine-readable config files** (MCP manifests, AI plugin manifests, skills.json)
- **Identity endpoints** (WebFinger, DID, Nostr NIP-05, AT Protocol)

### What we don't do

- **No crawling.** We don't follow links or discover new URLs.
- **No content indexing.** We never read, store, or index page content.
- **No authentication bypass.** We only check publicly accessible endpoints.
- **No continuous scanning.** Every scan is user-initiated. No autonomous traversal.

## Data we store

We store scan **metadata only**: the score, which capabilities were found, and which payment protocols were detected. We do not store page content, response bodies, or any sensitive data from the scanned site.

## Rate limits

- Each domain can only be scanned **once per hour**
- Individual users are rate-limited to prevent abuse
- Each scan sends ~15-20 requests over ~10 seconds

## Our User-Agent

All requests from agentgrade identify themselves with:

```
agentgrade/0.2 (+https://agentgrade.com/kb/about-scanning)
```

## Does agentgrade respect [robots.txt](/kb/robots-txt)?

robots.txt is a standard for search engine crawlers that systematically discover and index content. agentgrade is a site auditor, not a crawler — it doesn't index or store your content. We read your robots.txt to **grade** it, not to obey crawling directives. This is the same approach used by Lighthouse, SecurityHeaders.com, and SSL Labs.

## How to opt out

You can block agentgrade by filtering the User-Agent `agentgrade` in your firewall or WAF. However, since agentgrade only checks machine-facing configuration that you've intentionally made public, blocking it means you lose visibility into how agents perceive your site.

## Questions?

If you have questions about how agentgrade scans your site, open an issue on [GitHub](https://github.com/aluminumio/agentgrade).

## Related

- [SKILL.md](/kb/skills)
- [OpenAPI](/kb/openapi)
- [A2A](/kb/a2a)
- [WebMCP](/kb/webmcp)
- [llms.txt](/kb/llms-txt)