# agentgrade — full content > Scan any site for AI agent capabilities and payment protocols. Lighthouse-like reports for agentic services. > Source: https://agentgrade.com ## About agentgrade scans any URL for agent readiness — payment protocols (x402, MPP, L402), discovery files (llms.txt, sitemap, robots.txt), MCP endpoints, OpenAPI specs, identity (WebFinger, DID, Nostr), content negotiation, and infrastructure hygiene. Returns a 0–100 score, a letter grade, and per-check fix hints. Live at https://agentgrade.com. Free API at https://agentgrade.com/api/scan?url=https://example.com. MCP endpoint at https://agentgrade.com/mcp. CLI: npm i -g agentgrade-cli. ## Agent Readiness — the canonical guide **Agent Readiness: how websites get ready for AI agents** (https://agentgrade.com/agent-readiness) Agent readiness is how prepared a website is to be used by autonomous AI agents — software that browses, queries, and transacts on a user’s behalf without rendering pages. A site is agent-ready when its capabilities (data, search, payments, tools) are exposed through machine-readable protocols rather than buried in JavaScript and clickflows. As of May 2026, the top 100 websites average a 55% agent readiness score; 99% fail basic content negotiation. **Q: What is agent readiness?** A: Agent readiness is how prepared a website is for autonomous AI agents — software that browses, queries, and transacts on a user’s behalf. It measures whether a site’s capabilities are exposed through machine-readable protocols rather than locked behind JavaScript and clickflows. **Q: Why does agent readiness matter in 2026?** A: AI agents now route a measurable share of commercial traffic for major platforms. Adobe Analytics tracked a 4,700% year-over-year surge in generative-AI traffic to US shopping sites by July 2025, with a further 393% Q1 2026 increase. eMarketer forecasts AI platforms will process roughly 1.5% of US retail ecommerce in 2026 (~$20.9 billion, ~4× 2025), and Morgan Stanley projects $385 billion in agentic-commerce impact by 2030. Sites that aren’t agent-ready get bypassed in favor of sites that are. **Q: What is the difference between SEO and agent readiness?** A: SEO optimizes for human search-engine traffic — ranking pages so people find and click them. Agent readiness optimizes for non-human callers — exposing data and capabilities so agents can use them programmatically. They share some signals (sitemap, robots.txt, structured data) but the goals differ: SEO wants visibility, agent readiness wants addressability. **Q: Is agent readiness the same as AEO or GEO?** A: They overlap. AEO (Answer Engine Optimization) and GEO (Generative Engine Optimization) focus on getting cited inside LLM answers (ChatGPT, Perplexity, Google AI Overviews). Agent readiness is broader — it also covers agent-driven transactions, tool calls, and capability exposure. If you’re being cited by ChatGPT, that’s AEO; if an agent can complete a purchase on your site, that’s agent readiness. **Q: How do I measure my site’s agent readiness?** A: Run a scan at agentgrade.com. It checks 70+ signals across discovery, capability exposure, content negotiation, payment protocols, and trust/identity. You get a 0–100 score, a letter grade, and per-check fix hints. The API is at /api/scan and there is also a CLI: npm i -g agentgrade-cli. **Q: What is a typical agent readiness score?** A: The top 100 websites by traffic average 55% as of May 2026. Cursor leads at 82%. Major social platforms (Facebook, X, Instagram, LinkedIn, Reddit) score 0% because they block scanners or require auth. Grade distribution across the top-100: 0 A grades, 43 B grades, 35 C grades, 12 D grades, 10 F grades. **Q: What is the single fastest way to improve my score?** A: Add /llms.txt — a plain-text file at your domain root listing key pages with one-line summaries. 65% of top-100 sites are still missing it, and it takes about 10 minutes to write a good one. **Q: Will agent readiness affect my SEO ranking?** A: Indirectly, in some cases. Some of the same signals (sitemap, structured data, content negotiation) feed both. But agent readiness is not a Google ranking factor in the conventional sense. Its impact is on agent-driven referrals — LLM citations, agentic checkout, programmatic API discovery. **Q: Is there a single open standard for agent readiness?** A: No. Agent readiness is a composite of many emerging protocols (MCP, x402, llms.txt, OpenAPI, A2A, WebMCP, SKILL.md). agentgrade’s scoring model weights each based on adoption, criticality, and whether the protocol can be verified at scan time. ## Author Eoin Siegel — Creator of agentgrade. Eoin Siegel is the creator of agentgrade, the scanner that grades websites on how prepared they are for AI agents. He works on the protocols and standards that make sites discoverable, queryable, and transactable by autonomous agents — including x402, MCP, llms.txt, OpenAPI, A2A, and SKILL.md. Profile: https://agentgrade.com/humans/eoin-siegel. undefined ## Knowledge Base --- ### Getting Started — Make Your Site Agent-Ready > Quick checklist to make your site discoverable and usable by AI agents. Source: https://agentgrade.com/kb/getting-started ## The 5-minute checklist These four files make your site visible to AI agents. Each takes minutes to add and immediately improves your agentgrade score. ### 1. Add [llms.txt](/kb/llms-txt) Create `/llms.txt` — a plain-text description of your service for LLMs: ```markdown # Your Service Name > One-line description of what your service does. ## API GET /api/search?q=term — Search items POST /api/create — Create an item (requires auth) ## Example curl "https://yoursite.com/api/search?q=test" ``` **Why:** LLMs read this to understand what your service does and how to use it. Start the file with an H1 heading for a verified check. ### 2. Add [OpenAPI](/kb/openapi) spec Serve `/openapi.json` describing your API endpoints: ```json { "openapi": "3.0.0", "info": { "title": "Your API", "version": "1.0" }, "paths": { "/api/search": { "get": { "summary": "Search items", "parameters": [ { "name": "q", "in": "query", "schema": { "type": "string" } } ] } } } } ``` **Why:** Agents use this to know exactly what endpoints exist and how to call them. Most frameworks can auto-generate this. ### 3. Add [robots.txt](/kb/robots-txt) with agent directives Add agent-specific rules to `/robots.txt`: ``` User-agent: GPTBot Allow: / User-agent: Claude-Web Allow: / User-agent: Anthropic-AI Allow: / ``` **Why:** Signals that your site welcomes AI agents. Many sites block AI crawlers by default. ### 4. Add a health endpoint Serve `/health` returning JSON: ```json { "status": "ok" } ``` **Why:** Agents and monitoring tools use this to check if your service is up before making requests. ## Next steps Once the basics are in place, consider: - **MCP endpoint** — Let AI models call your tools directly via JSON-RPC 2.0 ([learn more](/kb/mcp)) - **Payment support** — Accept micropayments from agents via x402 or [MPP](/kb/mpp) ([learn more](/kb/x402)) - **Service catalog** — Publish discoverable endpoints via Bazaar ([learn more](/kb/bazaar)) - **Skills manifest** — Curate the key actions agents should take ([learn more](/kb/skills)) ## Scan your site Enter your domain on [agentgrade.com](https://agentgrade.com) to see your current score and what's missing. ## Related - [A2A](/kb/a2a) - [WebMCP](/kb/webmcp) --- ### Understanding Your Score > How agentgrade scores your site, what verified vs declared means, and how to improve. Source: https://agentgrade.com/kb/understanding-your-score ## How scoring works agentgrade checks for 10 capabilities across payments, discovery, and identity. Each capability has sub-checks that fall into two tiers: - **Verified** — agentgrade made a live request and confirmed the capability works (e.g., [MCP](/kb/mcp) handshake completed, [OpenAPI](/kb/openapi) spec returned valid JSON) - **Declared** — a manifest or config file was found that claims the capability exists, but it wasn't independently confirmed Verified checks are weighted **2x** in the score calculation. A site that works is worth more than a site that claims to work. ## The 10 capabilities ### Payments (highest weight) - **Machine Payments** — [x402](/kb/x402), [MPP](/kb/mpp), or [L402](/kb/l402) detected via HTTP 402 headers - **Bazaar** — x402.json service catalog with discoverable endpoints ### Discovery - **MCP** — Model Context Protocol endpoint (JSON-RPC 2.0) - **AI Plugin** — ai-plugin.json manifest with reachable API spec - **Claude Plugin** — claude.json manifest with valid endpoints - **OpenAPI** — openapi.json spec that returns valid JSON - **[llms.txt](/kb/llms-txt)** — LLM context file starting with an H1 heading - **Skills** — skills.json manifest with reachable endpoints ### Informational (lower weight) - **[agents.txt](/kb/agents-txt)** — Agent access policy file - **[robots.txt](/kb/robots-txt)** — Agent-aware directives for AI crawlers ### Optional (don't hurt your score) - **Identity** — WebFinger, DID, Nostr NIP-05, AT Protocol DID ## Score breakdown The percentage score is: **(passed check weight / total check weight) × 100** The summary shows "X / Y checks passed" where Y excludes optional checks that weren't found. Optional checks (identity protocols) can only help your score, never hurt it. ## How to improve your score 1. **Add discovery files first** — `/openapi.json` and `/llms.txt` are the easiest wins 2. **Make sure they work** — a reachable, valid OpenAPI spec gets you verified checks, not just declared 3. **Add MCP** — the MCP handshake (initialize → tools/list → tools/call) earns multiple verified checks 4. **Add payment support** — x402 or MPP headers on at least one endpoint lights up the Machine Payments capability 5. **Publish a service catalog** — `/.well-known/x402.json` with discoverable endpoints earns Bazaar checks ## Related - [SKILL.md](/kb/skills) - [A2A](/kb/a2a) - [WebMCP](/kb/webmcp) --- ### Cloudflare & Agent Access > Why Cloudflare blocks agent traffic and how to configure it for agent accessibility. Source: https://agentgrade.com/kb/cloudflare-agent-access ## The problem Cloudflare's managed challenge (bot protection) blocks automated HTTP requests — including those from AI agents. If your site is behind Cloudflare with default settings, agents can't reach your API endpoints, payment gates, or discovery files. agentgrade detects this per-probe. When a check is blocked, you'll see "Blocked by Cloudflare" instead of a pass/fail result. Your score may appear low not because capabilities are missing, but because Cloudflare prevented verification. ## What gets blocked Cloudflare's challenge triggers on requests that: - Don't execute JavaScript (all API/agent requests) - Lack browser fingerprints (TLS, headers, cookies) - Come from cloud/datacenter IP ranges This means **every agent-facing endpoint** is potentially blocked: [MCP](/kb/mcp), [OpenAPI](/kb/openapi), [llms.txt](/kb/llms-txt), x402.json, payment gates, etc. ## How to fix it ### Option 1: Bypass challenge for API paths (recommended) In Cloudflare's WAF rules, create a rule that skips the managed challenge for agent-facing paths: **Expression:** ``` (http.request.uri.path matches "^/\.well-known/.*" or http.request.uri.path matches "^/api/.*" or http.request.uri.path eq "/mcp" or http.request.uri.path eq "/openapi.json" or http.request.uri.path eq "/llms.txt" or http.request.uri.path eq "/agents.txt" or http.request.uri.path eq "/skills.json" or http.request.uri.path eq "/robots.txt") ``` **Action:** Skip → Managed Challenge This keeps bot protection on your HTML pages while allowing agents to reach machine-facing endpoints. ### Option 2: Allowlist specific User-Agents Create a WAF rule that skips the challenge for known agent User-Agents: ``` (http.user_agent contains "agentgrade" or http.user_agent contains "Claude" or http.user_agent contains "GPTBot") ``` ### Option 3: Use Cloudflare's Bot Management tiers Cloudflare Enterprise and Business plans offer more granular bot management that can distinguish between "good bots" (API clients, agents) and malicious bots. ## Verifying the fix After updating your Cloudflare rules, scan your site again on agentgrade. Previously blocked checks should now show pass/fail results instead of "Blocked by Cloudflare." ## Related - [WebMCP](/kb/webmcp) - [SKILL.md](/kb/skills) - [A2A](/kb/a2a) --- ### Comparing Payment Protocols > x402 vs MPP vs L402 — when to use each machine payment protocol. Source: https://agentgrade.com/kb/comparing-payment-protocols ## Overview Three payment protocols let AI agents pay for API access. Each has different trade-offs: | | [x402](/kb/x402) | [MPP](/kb/mpp) | [L402](/kb/l402) | |---|---|---|---| | **Network** | Base (Ethereum L2) | Tempo | Bitcoin Lightning | | **Currency** | USDC | pathUSD | BTC (satoshis) | | **Speed** | ~2 seconds | Sub-second | Milliseconds | | **Fees** | Low (L2 gas) | Very low | Very low | | **SDK** | @coinbase/x402-express | mppx | Custom (LND/CLN) | | **Facilitator** | Coinbase | Tempo network | Lightning node | | **Stablecoin** | Yes (USDC) | Yes (pathUSD) | No (BTC volatility) | ## When to use x402 **Best for:** Production services that want the broadest ecosystem support. - Backed by Coinbase with official SDKs in TypeScript, Go, and Python - USDC stablecoin means predictable pricing - Base L2 keeps gas fees low - Largest growing ecosystem of x402-enabled services - Includes Bazaar discovery for service catalogs ## When to use MPP **Best for:** High-frequency micropayments where speed matters. - Sub-second finality on the Tempo chain - Designed specifically for machine-to-machine payments - pathUSD stablecoin (1:1 with USD) - Simple SDK (mppx) with Express middleware - Lowest overhead per transaction ## When to use L402 **Best for:** Bitcoin-native services and Lightning ecosystem integration. - Instant settlement via Lightning Network - No stablecoin — payments in satoshis (BTC) - Macaroon-based auth tokens enable flexible access control - Established ecosystem (since 2020, originally "LSAT") - Requires running or connecting to a Lightning node ## Can I support multiple protocols? Yes. agentgrade checks for all three independently. Supporting multiple protocols gives agents more payment options. The agent chooses which to use based on what wallets/funds it has available. Publish all supported protocols in your `/.well-known/x402.json` service catalog so agents can discover them. ## Which should I start with? If you're unsure, **start with x402**. It has the most complete SDK, Coinbase backing, and stablecoin pricing. Add MPP or L402 later if your use case benefits from their specific strengths. ## Related - [OpenAPI](/kb/openapi) --- ### About Scanning > How agentgrade scans sites, what data it collects, and how to opt out. Source: https://agentgrade.com/kb/about-scanning ## How agentgrade scans your site agentgrade is a diagnostic tool — like SSL Labs for agent-readiness. When someone enters a domain, we send a small set of targeted HTTP requests to check what agent-facing capabilities the site exposes. ### What we check - **~15-20 requests** to well-known paths (`/.well-known/x402.json`, `/openapi.json`, `/robots.txt`, etc.) - **HTTP headers** on responses (Payment-Required, WWW-Authenticate, Content-Type) - **Machine-readable config files** (MCP manifests, AI plugin manifests, skills.json) - **Identity endpoints** (WebFinger, DID, Nostr NIP-05, AT Protocol) ### What we don't do - **No crawling.** We don't follow links or discover new URLs. - **No content indexing.** We never read, store, or index page content. - **No authentication bypass.** We only check publicly accessible endpoints. - **No continuous scanning.** Every scan is user-initiated. No autonomous traversal. ## Data we store We store scan **metadata only**: the score, which capabilities were found, and which payment protocols were detected. We do not store page content, response bodies, or any sensitive data from the scanned site. ## Rate limits - Each domain can only be scanned **once per hour** - Individual users are rate-limited to prevent abuse - Each scan sends ~15-20 requests over ~10 seconds ## Our User-Agent All requests from agentgrade identify themselves with: ``` agentgrade/0.2 (+https://agentgrade.com/kb/about-scanning) ``` ## Does agentgrade respect [robots.txt](/kb/robots-txt)? robots.txt is a standard for search engine crawlers that systematically discover and index content. agentgrade is a site auditor, not a crawler — it doesn't index or store your content. We read your robots.txt to **grade** it, not to obey crawling directives. This is the same approach used by Lighthouse, SecurityHeaders.com, and SSL Labs. ## How to opt out You can block agentgrade by filtering the User-Agent `agentgrade` in your firewall or WAF. However, since agentgrade only checks machine-facing configuration that you've intentionally made public, blocking it means you lose visibility into how agents perceive your site. ## Questions? If you have questions about how agentgrade scans your site, open an issue on [GitHub](https://github.com/aluminumio/agentgrade). ## Related - [SKILL.md](/kb/skills) - [OpenAPI](/kb/openapi) - [A2A](/kb/a2a) - [WebMCP](/kb/webmcp) - [llms.txt](/kb/llms-txt) --- ### x402 — HTTP Payment Protocol > The HTTP 402 payment protocol that lets agents pay per-request with USDC on Base. Specification, comparison with traditional API monetization, ecosystem, and how to implement. Source: https://agentgrade.com/kb/x402 ## What is x402? x402 turns the HTTP 402 `Payment Required` status code into a real payment protocol. When an agent hits a paid endpoint, the server returns a 402 response with a `Payment-Required` header containing structured payment instructions. The agent (or its wallet) inspects the header, decides whether the price is acceptable, pays on-chain, attaches the receipt to a retry, and receives the response. Conceptually, x402 finally completes the HTTP 402 status code that has sat unused in the HTTP spec since 1996 — the spec reserved the code for "Payment Required" but never specified *how* a client should pay. x402 specifies the mechanism: USDC on Base, signed payment intents, facilitator-mediated settlement, and a discovery format so agents can find paid endpoints before calling them. ## Why x402 matters Traditional API monetization is built for humans: sign up, generate an API key, manage a credit card on file, get a monthly invoice. Every step requires a UI, an account, and an out-of-band relationship between the API provider and the developer. Autonomous agents don't have those things. An agent calling a paid API for the first time needs to: - Discover the price *before* committing - Pay programmatically, without human approval - Settle the transaction in seconds, not days - Receive a receipt the API provider can verify x402 addresses all four with a single mechanism. The price is in the 402 header. The payment is a signed transaction on Base. Settlement is sub-second. The receipt is the on-chain confirmation. This makes x402 the foundational payment primitive for the agentic web — analogous to what TLS was for the commercial web, or OAuth for delegated authorization: a protocol that turns something previously human-mediated into something machines can negotiate end-to-end. ## How it works 1. Agent sends a request to a paid endpoint 2. Server returns **HTTP 402** with a base64-encoded `Payment-Required` header 3. The header contains: amount, asset (USDC), network (Base), recipient wallet, and facilitator 4. Agent pays via the facilitator (e.g., Coinbase) 5. Agent retries the request with a `Payment` header containing the receipt 6. Server verifies the payment through the facilitator and serves the response The whole flow takes a few seconds end-to-end. No API keys. No subscriptions. No human in the loop. ## x402 vs traditional API monetization | Aspect | API keys + subscription | x402 | |---|---|---| | Signup | Account creation required | None | | Authentication | Per-account API key | Per-request payment | | Billing model | Monthly subscription | Pay-per-request | | Discovery | Out-of-band (docs, sales) | In-protocol (402 header) | | Settlement | Invoice cycle (~30 days) | Sub-second (on-chain) | | Failure mode | Quota exceeded → 429 | Payment insufficient → 402 | | Agent compatibility | Hard (account management) | Native | | Geographic limits | Common (Stripe restrictions) | Minimal (on-chain) | The key shift: x402 puts pricing into the protocol itself, where any agent can read it. With API keys, the price exists only in a contract negotiated between humans. With x402, the price is in the HTTP response, the same way the status code is — first-class transport metadata, not paperwork. ## Key concepts - **Facilitator**: A service that verifies and settles payments. Set `"facilitator": "coinbase"` in your x402.json to use Coinbase's facilitator on Base mainnet. Other facilitators can support different networks and assets. - **Envelope v2**: The current header format. Adds structured service catalogs, Bazaar discovery, and richer payment options. Always use v2. - **Network**: Payments happen on Base (`eip155:8453`) using USDC. The USDC contract on Base is `0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913`. Multi-network support is on the roadmap. - **Per-request pricing**: Each API call has its own price — no subscriptions, no tiered plans, no API keys. - **Discoverable services**: Mark services with `"discoverable": true` to appear in the [x402 Bazaar](/kb/bazaar) — a catalog of paid endpoints that agents browse to find capabilities. ## Who's using x402 The x402 ecosystem has formed around two anchors: - **Coinbase Developer Platform** — the protocol originator. Ships official SDKs in TypeScript, Go, and Python; runs the canonical facilitator on Base mainnet. - **Tempo [MPP](/kb/mpp)** — the Paradigm and Stripe-backed payment proxy. Tempo runs `*.mpp.tempo.xyz` proxies that translate between x402 and traditional payment rails for sites that haven't natively implemented x402. Beyond those anchors, individual API providers, agent runtimes, and [MCP](/kb/mcp) server operators are integrating x402 for per-call billing — paid MCP tools, metered data APIs, AI inference endpoints, and agent marketplaces. agentgrade's x402 Bazaar discovery catalogs paid endpoints across the agentic web as they go live. ## How to add x402 to your service ### 1. Install the SDK ```bash npm install @coinbase/x402-express ``` ### 2. Add middleware to your Express app ```javascript import { paymentMiddleware } from '@coinbase/x402-express'; app.use('/api/paid-endpoint', paymentMiddleware({ payTo: '0xYourWalletAddress', amount: '100000', // 0.10 USDC (smallest unit) network: 'base', facilitator: 'coinbase', })); ``` ### 3. Publish discovery file Serve `/.well-known/x402.json` so agents can find your paid endpoints: ```json { "x402Version": 2, "name": "Your Service", "network": "base", "facilitator": "coinbase", "payTo": "0xYourWalletAddress", "services": [{ "method": "POST", "path": "/api/paid-endpoint", "amount": "100000", "discoverable": true }] } ``` ### 4. Test the flow ```bash curl -i https://your-domain.com/api/paid-endpoint # Expect: HTTP/1.1 402 Payment Required # Expect: Payment-Required: ``` Decode the `Payment-Required` header (base64 → JSON) and confirm amount, asset, recipient, and facilitator match your middleware configuration. ## Common errors and debugging - **402 with no Payment-Required header**: Middleware not configured. Confirm `payTo` and `amount` are set on the middleware call. - **Payment header rejected**: Receipt is for the wrong amount, wrong asset, or expired. Most facilitators reject receipts older than a few minutes. - **Wallet address mismatch**: `payTo` in middleware doesn't match the receipt's recipient. Must match exactly. - **Network mismatch**: Receipt is on a different chain than declared. Confirm `network` is `base` in both middleware and x402.json. - **Facilitator unreachable**: The facilitator service is down. Use a fallback facilitator or fail open during outages. agentgrade's scanner probes these directly — its x402 check verifies live 402 responses parse cleanly and that the discovery file matches what endpoints actually return. ## Frequently asked questions ### Is x402 a blockchain protocol or an HTTP protocol? Both. The transport is HTTP (a 402 status with structured headers). The settlement is on-chain (USDC on Base). x402 is the bridge — the protocol that lets HTTP servers and HTTP clients negotiate a payment that gets settled by a blockchain. ### Do I need a crypto wallet to accept x402 payments? Yes. You need a Base wallet to receive USDC. Coinbase's CDP makes this manageable for non-crypto-native teams — you can provision a custodied wallet via API and never touch keys yourself. ### What's the difference between x402 and Stripe? Stripe is the dominant traditional rail: per-account API keys, monthly subscriptions, fraud detection trained on human buyers. x402 is per-request, agent-native, and settles on-chain. They're complementary — Stripe for human checkout, x402 for agent metered access. Tempo MPP bridges between them for sites that want both. ### Is x402 production-ready? Yes. x402 v2 is supported by Coinbase with official SDKs in TypeScript, Go, and Python. The canonical facilitator runs in production and the ecosystem of paid services is growing. ### Can agents pay each other with x402, not just API providers? Yes. x402 doesn't care whether the "server" is a human-run API, an agent, or an MCP tool. Any HTTP responder can return a 402 and any HTTP requester can pay. Agent-to-agent metered work is one of the most active use cases. ### Why USDC and not ETH? Stable. Predictable price. Most API providers price in dollars and want a dollar-stable asset for settlement; volatile assets create accounting friction that defeats the per-request model. ### What happens if the agent doesn't have funds? The wallet returns insufficient-funds and the call fails. There's no retry-buy-credits flow in the protocol itself — funding is the wallet's concern, not the API's. ### How discoverable are x402 endpoints? Endpoints marked `"discoverable": true` in your x402.json appear in the x402 Bazaar — the public catalog of paid services. Agents browse the Bazaar the way developers browse npm. agentgrade's scanner walks the Bazaar to surface paid endpoints across the agentic web. ## Spec maturity **Production-ready.** x402 v2 is the current envelope, supported by Coinbase with official SDKs in TypeScript, Go, and Python. The protocol is stable, the canonical facilitator runs in production, and the ecosystem is expanding across MCP tool providers, data APIs, and agent marketplaces. ## Learn more - [x402.org](https://www.x402.org/) — Protocol specification and whitepaper - [Coinbase x402 docs](https://docs.cdp.coinbase.com/x402/docs/welcome) — SDK and integration guide - [x402 npm package](https://www.npmjs.com/package/@coinbase/x402-express) — Express middleware - [Agent Readiness](/agent-readiness) — How x402 fits in the broader agent-readiness landscape ## Related - [OpenAPI](/kb/openapi) - [A2A](/kb/a2a) --- ### MPP — Machine Payment Protocol > Micropayment protocol using pathUSD on the Tempo network for sub-second agent payments. Source: https://agentgrade.com/kb/mpp ## What is MPP? MPP (Machine Payment Protocol) is a micropayment system built for AI agents. It uses pathUSD on the Tempo blockchain for near-instant, low-fee payments. Agents pay per API call using the mppx SDK. ## How it works 1. Agent sends a request to a paid endpoint 2. Server returns **HTTP 402** with a `WWW-Authenticate: Payment` header 3. The header contains: realm, amount, recipient, and network details 4. Agent pays using the mppx SDK 5. Agent retries with payment proof 6. Server verifies and responds ## Key concepts - **pathUSD**: A stablecoin on the Tempo chain (1 pathUSD = 1 USD) - **mppx**: The CLI and SDK for making and receiving MPP payments - **Sub-second finality**: Payments confirm almost instantly - **Low fees**: Designed for micropayments (fractions of a cent) ## How to add MPP to your service ### 1. Install mppx ```bash npm install mppx ``` ### 2. Create a wallet ```bash npx mppx account create npx mppx account fund # for testnet ``` ### 3. Add Express middleware ```javascript import { mppMiddleware } from 'mppx'; app.use('/api/paid-endpoint', mppMiddleware({ amount: 0.01, // in USD })); ``` ### 4. Set environment variables ```bash MPP_SECRET_KEY=your_secret_key MPP_RECIPIENT_ADDRESS=your_wallet_address ``` ## Spec maturity **Production-ready.** Live on the Tempo network with active services including AgentNews. ## Learn more - [mppx on npm](https://www.npmjs.com/package/mppx) — SDK, CLI, and Express middleware ## Related - [OpenAPI](/kb/openapi) --- ### SPT — Stripe Shared Payment Tokens > Stripe card payments for AI agents via Shared Payment Tokens and the MPP 402 flow. Source: https://agentgrade.com/kb/spt ## What is SPT? SPT (Shared Payment Token) lets AI agents pay for API calls with Stripe — using real card payments instead of crypto. A user pre-authorizes a token with spending limits, and the agent uses it to pay services autonomously. ## How it works 1. User creates an SPT via Stripe with spending limits (max amount, currency, expiration) 2. Agent sends a request to a paid endpoint 3. Server returns **HTTP 402** with `WWW-Authenticate: Payment method="stripe", intent="charge", request=""` 4. Agent decodes the `request` field to learn the price (amount, currency) 5. Agent retries with `Authorization: Payment ` containing the SPT 6. Server creates a Stripe PaymentIntent using the token, verifies it succeeded, and responds ## Key concepts - **Shared Payment Token**: A scoped, limited-use Stripe token (prefixed `spt_`) that grants an agent permission to spend - **Usage limits**: Each token has max amount, currency, and expiration — the user controls how much the agent can spend - **mppx middleware**: The same middleware that handles [MPP](/kb/mpp) (Tempo) payments also handles SPT — a server can accept both - **Multiple challenges**: A 402 response can include multiple `WWW-Authenticate: Payment` headers for different methods (tempo, stripe), letting agents choose how to pay ## How agents discover SPT SPT uses two discovery channels: 1. **Runtime (authoritative)**: The HTTP 402 response with `method="stripe"` in the `WWW-Authenticate: Payment` challenge 2. **Pre-request (advisory)**: [OpenAPI](/kb/openapi) `x-payment-info` extension on operations that require payment The 402 response is always authoritative — OpenAPI metadata is advisory only. ## How to add SPT to your service 1. Install mppx: `npm install mppx` 2. Configure your Stripe secret key 3. The mppx middleware will return 402 challenges for both Tempo and Stripe methods 4. Agents with an SPT can pay via Stripe; agents with pathUSD can pay via Tempo ## Learn more - [Stripe SPT docs](https://docs.stripe.com/agentic-commerce/concepts/shared-payment-tokens) — Token creation and usage - [mppx on npm](https://www.npmjs.com/package/mppx) — Middleware that handles both MPP and SPT - [paymentauth.org](https://paymentauth.org) — Payment discovery spec --- ### How AI Agents Actually Browse the Web > How ChatGPT, Perplexity, and Gemini retrieve and process web content — and what it means for your site. Source: https://agentgrade.com/kb/how-ai-agents-browse ## The problem with assumptions Most website optimization assumes a visitor fetches your page via HTTP, sees the headers, follows redirects, and renders HTML. AI agents break every one of these assumptions. ## ChatGPT ChatGPT's browsing tool fetches pages live via HTTP, but the model never sees the raw response: - **Text extraction only** — HTML is stripped to ~4,096 tokens of plain text before the model sees it - **No headers** — the model never knows Content-Type, status codes, or redirects - **SearchGPT intermediate** — a secondary model checks for prompt injection before content reaches the main model - **Agent Mode** uses a fake Chrome UA (`Chrome/138.0.0.0`) and identifies via RFC 9421 cryptographic signatures, not User-Agent **What this means:** Content negotiation works silently (the tool layer handles it), but the model only sees the extracted text. Serve clean, structured text and your content will be more useful to ChatGPT. ## Perplexity Perplexity uses a multi-stage retrieval pipeline: - **Stealth crawlers** — 3-6 million requests/day with generic Chrome UAs and rotating IPs, not `PerplexityBot` - **Hybrid ranking** — BM25 keyword matching + vector similarity to find relevant passages - **Atomic span retrieval** — extracts specific text spans rather than full pages - **Separate index** — maintains its own crawled index alongside web search results **What this means:** Your `robots.txt` rules for `PerplexityBot` may not stop their stealth crawlers. Structured content with clear headings helps their span extraction find the right passages. ## Gemini Gemini's most common browsing mode never hits your server at all: - **Index-based** — `url_context` reads from Google's internal index, not live HTTP. When tested, no request appeared in server logs - **Screenshot-based** — Project Mariner renders the page visually for tasks that need it - **Rejected markdown** — Gemini CLI rejected `Accept: text/markdown` responses in early testing **What this means:** Your site needs to be indexed by Googlebot for Gemini to see it. Adding `` in your HTML ensures Google indexes the [llms.txt](/kb/llms-txt) relationship. JSON-LD structured data also survives the indexing pipeline. ## What to do about it | Action | Helps with | |--------|------------| | Serve `llms.txt` with clean markdown | ChatGPT, Perplexity | | Add `` | Gemini (via Google index) | | Add JSON-LD structured data | Gemini (via Google index) | | Don't block `Google-Extended` in [robots.txt](/kb/robots-txt) | Gemini | | Use RFC 9421 signatures for bot auth | ChatGPT Agent Mode verification | | Serve structured content with clear headings | Perplexity span extraction | ## Learn more - [Dejan.ai: Google's URL Context Tool](https://dejan.ai/blog/googles-new-url-context-tool/) - [Dejan.ai: AI Mode Is Not Live Web](https://dejan.ai/blog/ai-mode-is-not-live-web/) - [Cloudflare: Perplexity Stealth Crawlers](https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/) - [SeatGeek: Chasing Signature](https://chairnerd.seatgeek.com/chasing-signature/) ## Related - [SKILL.md](/kb/skills) - [OpenAPI](/kb/openapi) - [A2A](/kb/a2a) - [WebMCP](/kb/webmcp) --- ### L402 — Lightning Payments > Bitcoin Lightning Network payments for API access using macaroons and invoices. Source: https://agentgrade.com/kb/l402 ## What is L402? L402 (formerly LSAT) uses Bitcoin's Lightning Network for instant API payments. The server issues a macaroon (a bearer token) paired with a Lightning invoice — once the agent pays the invoice, the macaroon becomes valid. ## How it works 1. Agent sends a request to a paid endpoint 2. Server returns **HTTP 402** with `WWW-Authenticate: L402 macaroon="...", invoice="..."` 3. Agent pays the Lightning invoice 4. Agent retries with the paid macaroon as proof (`Authorization: L402 macaroon:preimage`) 5. Server verifies the macaroon and responds ## Key concepts - **Macaroon**: A cryptographic bearer token that becomes valid once its paired invoice is paid - **Lightning invoice**: A one-time BOLT11 payment request on Bitcoin's Lightning Network - **Preimage**: The proof-of-payment revealed when a Lightning invoice is settled - **Instant settlement**: Lightning payments confirm in milliseconds - **Bitcoin-native**: Payments are in satoshis (1 sat = 0.00000001 BTC) ## How to add L402 to your service 1. Run a Lightning node (LND, CLN) or use a Lightning Service Provider 2. For each paid request, generate a macaroon + Lightning invoice pair 3. Return 402 with `WWW-Authenticate: L402 macaroon="", invoice=""` 4. On retry, verify the macaroon and check the invoice preimage ## Spec maturity **Established.** Originally developed by Lightning Labs as LSAT, renamed to L402. Used in production by multiple Lightning-powered APIs. ## Learn more - [L402 spec](https://docs.lightning.engineering/the-lightning-network/l402) — Lightning Labs documentation - [Aperture](https://github.com/lightninglabs/aperture) — L402 reverse proxy by Lightning Labs ## Related - [OpenAPI](/kb/openapi) --- ### MCP — Model Context Protocol > The open standard for connecting AI assistants to external tools. Spec, transport options, ecosystem, authentication, debugging, and how to implement an MCP server. Source: https://agentgrade.com/kb/mcp ## What is MCP? MCP (Model Context Protocol) is the open standard for connecting AI assistants to external tools, data sources, and services. Your server exposes "tools" — named functions with descriptions and input schemas — and the AI model decides when to call them mid-conversation. Created by Anthropic in late 2024 and now supported by Claude, ChatGPT, IDE plugins, and a growing list of agent frameworks, MCP plays the same role for AI that USB plays for hardware: one cable, any device. ## Why MCP matters Before MCP, every AI assistant connected to every external service through bespoke "plugin" or "function calling" code. Each AI vendor had its own format. Each tool integration had to be re-written per vendor — once for Claude, again for ChatGPT, again for Cursor, again for a custom agent. MCP standardizes the interface. A single MCP server works with every MCP-compatible client. That means: - Tool builders write the integration **once** instead of N times - AI clients pick up new tools the moment they're published, no client update required - Agents can discover, list, and call tools at runtime — no compile-time wiring - Open-source servers can be audited, forked, and self-hosted It's the difference between proprietary printer drivers and the IP standard. The protocol is boring. Standardizing on it unlocks an ecosystem. ## How it works All communication uses **JSON-RPC 2.0** over HTTP POST to a single endpoint (e.g., `/mcp`): 1. Client sends `initialize` to establish the session 2. Server responds with its name, version, and capabilities 3. Client sends `tools/list` to discover available tools 4. Server returns a list of tools with descriptions and input schemas 5. Client sends `tools/call` with a tool name and arguments 6. Server executes the tool and returns the result ## MCP vs alternatives | Aspect | OpenAI function calling | [OpenAPI](/kb/openapi) / Swagger | MCP | |---|---|---|---| | Vendor scope | Single vendor | Vendor-neutral | Vendor-neutral | | Designed for | LLM tool calls | REST APIs | AI agents | | Discovery | Hardcoded in client | Static spec file | Runtime via `tools/list` | | Streaming | Limited | None native | Built-in (SSE / Streamable HTTP) | | Stateful sessions | No | No | Yes (initialize → calls) | | Auth | Client-side keys | API keys, OAuth | OAuth, bearer, custom | | Ecosystem | OpenAI only | Mature, broad | New, growing fast | OpenAPI describes a REST API at design time. MCP describes a *live* tool surface at runtime. The two are complementary — many MCP servers wrap existing OpenAPI services and expose them as agent-callable tools. ## Implementation example ```javascript app.post('/mcp', (req, res) => { const { method, params, id } = req.body; if (method === 'initialize') { return res.json({ jsonrpc: '2.0', id, result: { protocolVersion: '2024-11-05', capabilities: { tools: {} }, serverInfo: { name: 'my-service', version: '1.0' } }}); } if (method === 'tools/list') { return res.json({ jsonrpc: '2.0', id, result: { tools: [{ name: 'search', description: 'Search for items', inputSchema: { type: 'object', properties: { query: { type: 'string' } }, required: ['query'] } }] }}); } if (method === 'tools/call') { const { name, arguments: args } = params; return res.json({ jsonrpc: '2.0', id, result: { content: [{ type: 'text', text: 'Search results...' }] }}); } res.json({ jsonrpc: '2.0', id, error: { code: -32601, message: 'Method not found' }}); }); ``` ## Transport options MCP supports three transports — pick based on where your client runs: - **Streamable HTTP** *(recommended for remote servers)*: Plain HTTP POST to one endpoint. Server returns either a JSON response or an SSE stream for tools that emit progress. Works through CDNs, load balancers, and firewalls. This is what `claude.ai` web and most hosted clients expect. - **SSE (Server-Sent Events)**: Older transport using a long-lived GET for server→client and POST for client→server. Still supported but Streamable HTTP supersedes it for new servers. - **stdio**: Process pipes — the client launches your server as a subprocess and communicates via stdin/stdout. Used by Claude Desktop and CLI agents. Zero network, zero CORS, but the client must be able to run your code locally. If you want one MCP server to work with Claude Desktop *and* the Claude web app *and* ChatGPT *and* Cursor, ship Streamable HTTP. Almost everyone speaks it now. ## CORS — required for browser-based agents A growing class of MCP clients runs *in the browser*: Claude.ai web, ChatGPT browser, browser extensions, and embedded chat widgets. When JavaScript running on `claude.ai` calls `fetch('https://yourdomain.com/mcp')`, the browser does two things on its own: 1. It actually sends the request to your server. 2. It refuses to hand the response back to the JavaScript that asked for it — *unless* your server's response includes an `Access-Control-Allow-Origin` header that names `claude.ai` (or `*`). So without CORS headers, your server runs fine and returns 200, but the MCP client sitting in the browser tab sees a network error and can never read the body. This is the browser protecting users from one site silently making authenticated requests to another. Server-side clients like Claude Desktop or the MCP CLI aren't subject to this — there's no browser policing the response — which is why an endpoint can work in Desktop but fail in the web. To support browser-based MCP clients, send these headers on every `/mcp` response, and handle the OPTIONS preflight: ```javascript app.use('/mcp', (req, res, next) => { res.set('Access-Control-Allow-Origin', '*'); res.set('Access-Control-Allow-Methods', 'GET, POST, OPTIONS'); res.set('Access-Control-Allow-Headers', 'Content-Type, Accept, Authorization'); if (req.method === 'OPTIONS') return res.sendStatus(204); next(); }); ``` ### Wildcard vs specific origin `Access-Control-Allow-Origin: *` is fine for **public, read-only** MCP endpoints. If your MCP endpoint requires authentication or charges for tool calls, return a specific origin (or echo back the request's `Origin` header against an allowlist) and set `Access-Control-Allow-Credentials: true`. Wildcard origins cannot be used together with credentials per the CORS spec. ### Why preflight matters For a "simple" request — a plain `GET` or a `POST` with `Content-Type: text/plain` — the browser just sends it and only the response gets blocked. But MCP uses `POST` with `Content-Type: application/json`, which the browser considers risky enough to *check first*. So before sending the actual POST, the browser sends an `OPTIONS` request asking your server: "Will you accept a POST from this origin with these headers?" If your server doesn't respond to the OPTIONS with a 2xx status and the right CORS headers, the browser cancels the POST entirely — it never reaches your server. That's why CORS middleware has to handle both: the OPTIONS preflight, *and* the actual POST. ## Authentication For public read-only servers, no auth is fine. For private or paid tools, MCP supports: - **Bearer tokens** in the `Authorization` header (simplest) - **OAuth 2.1** with PKCE — the standard for user-delegated access; the MCP spec defines a metadata endpoint at `/.well-known/oauth-authorization-server` so clients can discover your OAuth flow automatically - **Per-request payment** via [x402](/kb/x402) — return HTTP 402 with payment instructions instead of authenticating in advance OAuth is the right default if humans are authorizing on behalf of themselves. Bearer tokens are right for service-to-service. x402 is right when calls are paid and stateless. ## Who's using MCP - **Anthropic Claude** — native MCP client in Claude Desktop, Claude.ai web, and the Claude API - **OpenAI ChatGPT** — added MCP support in 2025 - **IDE integrations** — Cursor, Zed, Windsurf, and JetBrains plugins all speak MCP - **Agent frameworks** — LangChain, LlamaIndex, and most agent SDKs include MCP transports - **Official MCP Registry** at `registry.modelcontextprotocol.io` lists hundreds of public servers - **Smithery** at `smithery.ai` is the largest community-maintained MCP marketplace agentgrade publishes its own MCP server (`com.agentgrade/mcp-server`) for agents that want to scan sites from inside an MCP-aware client. ## Common errors and debugging - **`Method not found`** (-32601): The client called a method your server doesn't implement. Always return a graceful error for unknown methods rather than throwing. - **CORS preflight fails**: OPTIONS returns 404 or 405. Add the middleware shown above — your server must handle OPTIONS as well as POST. - **Tools list is empty in the client UI**: Your `tools/list` response is well-formed but the client expects a non-empty `tools` array. Confirm at least one tool is registered. - **`Invalid params`** (-32602): The arguments sent by the client don't match your `inputSchema`. Look at the JSON Schema — required fields missing or wrong types are the usual culprits. - **Works in Desktop, fails in the web**: Almost always CORS. The browser blocks the response; server logs show 200s but the client sees nothing. Open the browser devtools network tab to confirm. ## Key concepts - **Tools**: Functions your server exposes (e.g., "search", "create_invoice") - **inputSchema**: JSON Schema describing what arguments each tool accepts - **Resources**: Read-only data your server can hand to the model (files, snippets, query results) - **Prompts**: Reusable prompt templates the server publishes for the client to use - **Streamable HTTP**: The recommended transport — all messages via POST to one endpoint - **SSE**: Server-Sent Events — older transport, kept for backward compatibility ## FAQ ### Is MCP an Anthropic protocol or an open standard? It was created by Anthropic and is published as an open specification. The reference implementations are MIT-licensed. OpenAI, IDE vendors, and agent frameworks have adopted it independently of Anthropic — it is a community standard now, not a single-vendor protocol. ### How is MCP different from OpenAI's function calling? Function calling is a feature of one vendor's API. MCP is a transport-level protocol any client can speak. With function calling, the model and tool definition live inside the same OpenAI request; with MCP, the model lives at the client and the tools live at your server, and they communicate over a defined wire format. ### Should I expose my existing REST API as MCP or just give the AI my OpenAPI spec? Both work, but they target different clients. Models with native MCP support (Claude, Cursor) call tools directly via MCP. Models that read OpenAPI specs need the spec rendered into their prompt or function-call format. If you only have time for one, ship MCP — most agent clients prefer it for runtime discovery. If you already have an OpenAPI spec, wrapping it in an MCP server is a few hundred lines. ### Does MCP require WebSockets? No. The current standard is Streamable HTTP — plain POST to one endpoint, with optional SSE streaming inside. Earlier drafts used WebSockets; that path was abandoned because plain HTTP traverses CDNs, load balancers, and corporate firewalls without configuration. ### Can MCP servers charge for tool calls? Yes. The protocol doesn't include payment, but you can layer x402 on top: have your server return HTTP 402 from `tools/call` until the agent attaches a valid payment receipt. This is one of the most active patterns in the agent ecosystem. ### How do I list my MCP server in the registry? Submit a manifest to [registry.modelcontextprotocol.io](https://registry.modelcontextprotocol.io). Smithery.ai mirrors the registry and adds discovery features. Listing is free and the review queue is short. ### Why is my MCP server slow? The protocol itself is cheap — JSON-RPC over HTTP. Slowness is almost always (a) the tool implementation, or (b) a cold serverless start. Profile the `tools/call` handler before blaming the transport. ### Do I need a separate server per tool, or one server with many tools? One server with many tools is the norm. Group tools by logical service (e.g., all GitHub operations in one server). Clients can connect to multiple servers simultaneously, so splitting only makes sense when permissioning or hosting needs differ. ## Spec maturity **Formal specification.** MCP is maintained by Anthropic with a versioned spec. Current protocol version: `2024-11-05`. Supported natively by Claude, ChatGPT, Cursor, and most agent frameworks. The Streamable HTTP transport is stable and used by every major client. ## Learn more - [modelcontextprotocol.io](https://modelcontextprotocol.io) — Official specification - [MCP TypeScript SDK](https://github.com/modelcontextprotocol/typescript-sdk) — Reference implementation - [Official MCP Registry](https://registry.modelcontextprotocol.io) — Public catalog of MCP servers - [Smithery](https://smithery.ai) — Largest community marketplace - [MDN: CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) — Cross-Origin Resource Sharing reference - [Agent readiness](/agent-readiness) — How MCP fits into the broader agentic web landscape ## Related - [SKILL.md](/kb/skills) - [A2A](/kb/a2a) - [WebMCP](/kb/webmcp) - [llms.txt](/kb/llms-txt) --- ### x-payment-info — OpenAPI Payment Discovery > Pre-request payment discovery via OpenAPI extensions so agents know pricing before triggering a 402. Source: https://agentgrade.com/kb/x-payment-info ## What is x-payment-info? `x-payment-info` is an [OpenAPI](/kb/openapi) extension that tells agents which endpoints cost money and how much — before they make a request. Instead of triggering a 402 to learn the price, agents can read the OpenAPI spec upfront. ## How it works Add the extension to any paid operation in your `/openapi.json`: ```json { "paths": { "/api/v1/submit": { "post": { "x-payment-info": { "authMode": "payment-required", "protocols": [ { "mpp": { "method": "tempo", "intent": "charge" } }, { "mpp": { "method": "stripe", "intent": "charge" } }, { "[x402](/kb/x402)": {} } ], "price": { "mode": "fixed", "amount": "0.50", "currency": "USD" } } } } } } ``` ## Key concepts - **Advisory, not authoritative**: The 402 response is always the source of truth. `x-payment-info` helps agents plan, but runtime challenges override it. - **Protocol-agnostic**: Lists all accepted payment methods (Stripe, Tempo, x402) so agents can choose. - **Pricing modes**: `fixed` (exact price) or `dynamic` (min/max range, e.g., bidding). - **Cache-friendly**: Recommended `Cache-Control: max-age=300` on the OpenAPI spec. ## Why it matters Without `x-payment-info`, agents must hit every endpoint to discover if it's paid and how much it costs. With it, agents can read the spec once and plan their budget across multiple API calls. ## How agentgrade checks for it agentgrade reads your `/openapi.json` and checks if any operation has an `x-payment-info` extension. This is an optional check in the OpenAPI scoring group — it can help your score but won't hurt it if missing. ## Learn more - [paymentauth.org spec](https://paymentauth.org/draft-payment-discovery-00.html) — Payment discovery draft specification --- ### Bazaar — Service Discovery > The x402 service catalog that lets agents discover and browse paid API endpoints. Source: https://agentgrade.com/kb/bazaar ## What is Bazaar? Bazaar is the discovery layer for [x402](/kb/x402). It answers: "What can I buy here, and how much does it cost?" Agents browse a structured catalog of services, prices, and input/output schemas at `/.well-known/x402.json`. ## How it works Publish `/.well-known/x402.json`: ```json { "x402Version": 2, "name": "Your Service", "description": "What your service does", "network": "base", "facilitator": "coinbase", "payTo": "0xYourWallet", "services": [ { "method": "POST", "path": "/api/generate", "description": "Generate content", "amount": "100000", "discoverable": true, "outputSchema": { "input": { "type": "http", "method": "POST", "bodyFields": { "prompt": { "type": "string", "required": true } } }, "output": { "type": "json", "schema": { "result": { "type": "string" } } } } } ] } ``` ## Required fields - **x402Version** — Spec version (use `2`; the scanner also accepts `1` — see below) - **name** — Provider name - **network** — Blockchain (e.g., `base`, or the CAIP form `eip155:8453` under v2) - **facilitator** — Settlement provider (e.g., `coinbase`) - **payTo** — Wallet address that receives payments - **services[]** — Array of paid endpoints with `method` and `path` ## v1 vs v2 x402 v2 launched in December 2025 and is the current spec, but v1 servers are still common in the wild. AgentGrade accepts both. Every protocol-level difference: - **Header names**: v1 used `X-`-prefixed headers (`X-PAYMENT` on the retry request, `X-PAYMENT-RESPONSE` on success). v2 drops the `X-` prefix (`PAYMENT-SIGNATURE`, `PAYMENT-RESPONSE`) — the `X-` convention was deprecated by [RFC 6648](https://www.rfc-editor.org/rfc/rfc6648) back in 2012. - **Where the challenge lives**: v1 put the payment challenge in the JSON *body* of the 402 response. v2 moves it into the `PAYMENT-REQUIRED` response header (base64-encoded JSON), freeing the body for a human-readable paywall page. - **`amount` field rename**: v1's `accepts[]` entries used `maxAmountRequired` for the price. v2 renames this to `amount`. - **`resource` shape**: v1 put `resource` (as a string URL), `description`, and `mimeType` inside each `accepts[]` entry. v2 hoists these into a top-level `resource` object with `url`, `description`, and `mimeType` fields — declared once for the whole 402 response instead of duplicated per accepts entry. - **Chain identification**: v1 used free-form strings like `"base"` for the `network` field. v2 standardizes on [CAIP](https://chainagnostic.org/) IDs like `eip155:8453`, so the same field works for non-EVM chains and off-chain rails. - **Dynamic `payTo`**: v1 declared one static recipient per service in the discovery catalog. v2 lets the server compute `payTo` per request — useful for marketplaces routing payments to individual sellers. - **Sessions**: v1 required the full payment dance on every call. v2 adds wallet-controlled sessions (Sign In With X402) — sign once, the server issues a session, subsequent calls skip the handshake. If you're building new, build v2. If you're auditing an existing v1 server, the scanner won't penalize you for being on v1 — but the v2 differences above are real efficiency wins. ## How to return a 402 The discovery catalog above declares *what* you sell. The 402 response is what actually triggers payment when an agent calls a paid endpoint without one. ### v2 form (current spec) The payment challenge goes in the `PAYMENT-REQUIRED` response header as a base64-encoded JSON object. The response body is yours to use however you want — typically a human-readable paywall page. The retry request arrives with the signed payment in the `PAYMENT-SIGNATURE` request header (also base64-encoded JSON). On success, the server may set a `PAYMENT-RESPONSE` header with settlement details. ```javascript app.post('/api/generate', async (req, res) => { if (!req.get('PAYMENT-SIGNATURE')) { const challenge = { x402Version: 2, resource: { url: 'https://yourdomain.com/api/generate', description: 'Generate content from a prompt', mimeType: 'application/json' }, accepts: [{ scheme: 'exact', network: 'base', // or CAIP form: 'eip155:8453' amount: '100000', // smallest units; 6-decimal USDC → $0.10 asset: '0x833589fCD6EDb6E08f4c7C32D4f71b54bdA02913', payTo: '0xYourWalletAddress', maxTimeoutSeconds: 60 }], extensions: { bazaar: { discoverable: true } } }; res.status(402); res.set('PAYMENT-REQUIRED', Buffer.from(JSON.stringify(challenge)).toString('base64')); return res.send('

$0.10 to generate content

'); } // Verify the signature with your facilitator, then run the work. // The @coinbase/x402 SDK handles the verification call for you. res.json({ result: '...' }); }); ``` ### v1 form (legacy, body-based) v1 servers put the challenge in the JSON *body* of the 402 response. The retry request used the `X-PAYMENT` header, and a successful settlement was returned in `X-PAYMENT-RESPONSE`. v1 also uses `maxAmountRequired` instead of v2's `amount`. ```javascript app.post('/api/generate', async (req, res) => { if (!req.get('X-PAYMENT')) { return res.status(402).json({ x402Version: 1, error: 'Payment required to access this resource', accepts: [{ scheme: 'exact', network: 'base', maxAmountRequired: '100000', asset: '0x833589fCD6EDb6E08f4c7C32D4f71b54bdA02913', payTo: '0xYourWalletAddress', resource: 'https://yourdomain.com/api/generate', maxTimeoutSeconds: 60 }] }); } // ...verify the X-PAYMENT signature, then serve the work }); ``` **Heads up — AgentGrade's live-402 check is v2-only.** Our scanner reads the `PAYMENT-REQUIRED` response header to confirm a working paywall. A v1 server that puts its challenge in the body will pass the discovery-catalog check (`/.well-known/x402.json`) but won't score on the live-402 check. If that matters to you, build v2. ### Use the SDK You almost certainly don't want to hand-roll signature verification — call the [`@coinbase/x402`](https://github.com/coinbase/x402) middleware instead. It wraps both response shapes and the facilitator round-trip in one line of Express/Fastify/Hono setup. ## extensions.bazaar in the live 402 header When an agent calls a paid endpoint without payment, your server returns HTTP 402 with a base64-encoded `Payment-Required` header. That header's JSON payload should declare `extensions.bazaar` so agents know this endpoint is part of a discoverable catalog. The JSON payload **before** base64 encoding (v2 form): ```json { "x402Version": 2, "accepts": [ { "scheme": "exact", "network": "base", "amount": "100000", "asset": "0xUSDC...", "payTo": "0xYourWallet", "maxTimeoutSeconds": 60 } ], "extensions": { "bazaar": { "discoverable": true } } } ``` Without `extensions.bazaar`, agents that hit a 402 have no signal that the endpoint is also catalogued at `/.well-known/x402.json` — they may treat it as a one-off paid endpoint instead of part of a browseable service. ## Spec maturity **Part of x402 v2.** Bazaar discovery is defined within the x402 specification. ## Learn more - [x402.org](https://www.x402.org/) — x402 protocol spec (includes Bazaar) ## Related - [OpenAPI](/kb/openapi) - [A2A](/kb/a2a) --- ### OpenAPI — API Specification > Machine-readable API description that tells agents what endpoints exist and how to call them. Source: https://agentgrade.com/kb/openapi ## What is OpenAPI? OpenAPI (originally Swagger) is the dominant specification for describing REST APIs in machine-readable form. An OpenAPI document — typically a JSON or YAML file at `/openapi.json` or `/swagger.json` — enumerates every endpoint, the HTTP verb, the parameters each accepts, the response schema, the authentication scheme, and any custom extensions. The 3.x version is maintained by the OpenAPI Initiative, a Linux Foundation project with contributors from Google, Microsoft, IBM, Postman, SmartBear, and others. For AI agents, OpenAPI is the difference between guessing your API surface and *knowing* it. An agent that can fetch your OpenAPI document has the same first-class affordance a generated SDK gives a human developer: every endpoint, parameter, type, and example, statically discoverable. ## Why OpenAPI matters for agents Agents calling REST APIs face an iteration cost problem. Without a schema, the loop is: guess the endpoint → call it → parse the error → guess the parameter shape → call it again → parse the next error. Three to five round-trips per endpoint is typical, and the agent burns tokens and clock time on each failed call. With OpenAPI, the agent can: - **Pick the right endpoint** by reading the `summary` and `description` fields against the user's intent. - **Construct a valid request the first time** by reading the parameter schema, including required fields, types, and enums. - **Parse responses correctly** by binding to the response schema rather than improvising regex. - **Discover authentication** from the `securitySchemes` object — including [x402](/kb/x402) payment requirements via the `x-payment-info` extension AgentGrade looks for. Agents that find OpenAPI on a site succeed on the first call at a measurably higher rate than agents that have to scrape docs. This is why a published OpenAPI spec is one of the highest-weighted checks AgentGrade runs on API-bearing sites. ## How OpenAPI works OpenAPI is a JSON or YAML document. It opens with metadata (`info`, `servers`), then enumerates every path under `paths`, then declares reusable schemas under `components`. Tools read it via HTTP fetch — no SDK or parser required beyond standard JSON handling. A minimal valid OpenAPI 3.0 document: ```json { "openapi": "3.0.3", "info": { "title": "Catalog API", "version": "1.0.0", "description": "Search the product catalog and place orders." }, "servers": [ { "url": "https://api.example.com" } ], "paths": { "/search": { "get": { "summary": "Search the product catalog", "parameters": [ { "name": "q", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Search query string" } ], "responses": { "200": { "description": "Search results", "content": { "application/json": { "schema": { "type": "array", "items": { "$ref": "#/components/schemas/Product" } } } } } } } } }, "components": { "schemas": { "Product": { "type": "object", "properties": { "id": { "type": "string" }, "title": { "type": "string" }, "price_usd": { "type": "number" } } } } } } ``` Most web frameworks generate this automatically from route definitions (FastAPI in Python, Hono in TypeScript, Echo in Go, Spring's springdoc-openapi in Java). Hand-authoring is rarely necessary. ## OpenAPI vs other agent-readability layers | Spec | Format | What it answers | Agent uses it for | |---|---|---|---| | **OpenAPI** | JSON/YAML | *Endpoints, parameters, types, auth* | Constructing valid requests | | **[SKILL.md](/kb/skills)** | Markdown + frontmatter | *How to use this service in prose* | Picking which endpoint to call | | **[MCP](/kb/mcp) manifest** | JSON-RPC | *Tools exposed over persistent connection* | Calling the server as a long-lived tool | | **[llms.txt](/kb/llms-txt)** | Markdown | *What is this site* | Triage — is this site relevant? | | **`ai-plugin.json`** | JSON | *Legacy ChatGPT plugin metadata* | Plugin discovery (deprecated) | OpenAPI is the schema-of-record. The other files refer to it; agents validate against it. A site with SKILL.md + OpenAPI gives an agent both the playbook and the formal contract. ## The `x-payment-info` extension OpenAPI supports custom extension fields prefixed with `x-`. AgentGrade looks for an `x-payment-info` object at the operation level that declares whether an endpoint is paid, the price, the protocol ([x402](/kb/x402), [L402](/kb/l402), [MPP](/kb/mpp), [SPT](/kb/spt)), and any payment URL. ```json { "paths": { "/api/order": { "post": { "summary": "Place an order", "x-payment-info": { "required": true, "protocol": "x402", "price_usd": "0.10", "currency": "USDC", "network": "base" } } } } } ``` This makes the paid-endpoint signal machine-checkable without the agent needing to issue a probe request and parse a 402 response. AgentGrade verifies the declared payment info matches a live probe — declared-only without live verification scores lower than declared + verified. ## Who's using OpenAPI OpenAPI is genuinely universal. Major API platforms (Stripe, Twilio, GitHub, Cloudflare, AWS via its OpenAPI generator) publish OpenAPI specs as the canonical reference. Major framework adoptions: - **FastAPI** (Python) — generates OpenAPI from type hints - **Hono** + `zod` — runtime-validated TypeScript routes that emit OpenAPI - **NestJS** (TypeScript) — decorator-driven OpenAPI generation - **Spring Boot** + springdoc — annotation-based OpenAPI for Java - **Echo** + echo-swagger — Go OpenAPI generation - **Rails** + rswag — Ruby OpenAPI generation For agents, the practical effect is that any modern API built in the last five years either has OpenAPI or can have it added with a single dependency. ## How to add OpenAPI to your service 1. **Pick a generator.** If your framework supports automatic OpenAPI generation, use it. Hand-writing is error-prone and goes stale. 2. **Serve it at a stable path.** `/openapi.json` is the conventional location; `/swagger.json` is accepted by most tools. Serving at the root or under `/.well-known/` are both fine. 3. **Set CORS.** Allow agent fetchers to read the document from a browser context: `Access-Control-Allow-Origin: *` on the spec endpoint. 4. **Declare `servers` with the production URL.** Without this, agents construct request URLs against the wrong origin. 5. **Add `description` fields.** Every endpoint, every parameter. Agents read these to pick endpoints and infer semantics. 6. **Declare `x-payment-info`** if any endpoint requires payment. Match it against the live response. 7. **Link from llms.txt or SKILL.md.** Optional but helps agents discover the spec. ## Common errors scanners flag - **Wrong content-type** — the spec is served as `text/html` (often a Swagger UI page) instead of `application/json`. Agents that look for the spec at `/openapi.json` get the UI HTML and fail to parse. - **Stale spec** — the file lists endpoints that no longer exist or omits new ones. Auto-generation prevents this. - **Missing `servers[].url`** — agents that read this field to construct request URLs default to the same origin as the spec, which is often wrong (e.g., spec on `docs.example.com`, API on `api.example.com`). - **No `description` on endpoints** — agents can read the path, but without prose descriptions they cannot reliably pick between similar endpoints. - **Declared `x-payment-info` doesn't match live probe** — claims an endpoint is paid but live request returns 200 without a payment challenge, or claims free but returns 402. - **CORS blocks the fetch** — the spec is served but a missing `Access-Control-Allow-Origin` header prevents browser-context agents from reading it. ## FAQ **What's the difference between OpenAPI and Swagger?** "Swagger" was the original name; in 2015 it was renamed to OpenAPI when donated to the Linux Foundation. SmartBear retains the "Swagger" brand for tools (Swagger UI, Swagger Editor). The spec is OpenAPI; the tooling is often still called Swagger. **Do I need OpenAPI if I have [SKILL.md](/kb/skills)?** Ship both if you have an API. SKILL.md is the "playbook" — prose, agent-readable. OpenAPI is the "contract" — formal, machine-validatable. Agents use SKILL.md to decide which endpoint to call and OpenAPI to construct the call correctly. **Should I serve at `/openapi.json` or `/.well-known/openapi.json`?** `/openapi.json` at the root is the dominant convention. Some agents probe `/.well-known/openapi.json` as a fallback. Serving both is fine. **OpenAPI 3.0 vs 3.1 vs 2.0 — does it matter?** 3.1 is current; 3.0 is fine and widely supported. 2.0 (the original "Swagger 2.0" format) is still recognized but you should migrate. Tools handle all three; agents typically don't care about the version line. **Can I serve OpenAPI for a GraphQL API?** OpenAPI describes REST. For GraphQL, agents introspect the schema directly via the GraphQL endpoint. Some sites publish both an OpenAPI shim and the GraphQL endpoint. **Does AgentGrade penalize sites with declared-only payment info?** Sites with OpenAPI declaring `x-payment-info` but no live verification score positively but below sites where the live probe confirms the declared protocol. Both are better than no signal. **What about `x-mcp-server` or other extensions?** OpenAPI's `x-` prefix is open. AgentGrade currently looks for `x-payment-info` on paid-endpoint declarations. Other agent-relevant extensions may be added as the ecosystem standardizes. **How big can an OpenAPI spec get before agents struggle?** Multi-MB specs are common at large API platforms (Stripe's is ~10MB). Agents generally fetch and parse fine but may not load the full spec into their context — they'll search or chunk. Keep `description` fields concise; that's where most token weight lands. ## Spec maturity **Universal industry standard.** OpenAPI 3.x is maintained by the OpenAPI Initiative under the Linux Foundation. Supported by virtually every API tool, code generator, AI coding assistant, and modern web framework. ## Learn more - [OpenAPI Specification](https://www.openapis.org/) — official spec and docs - [Swagger Editor](https://editor.swagger.io/) — write and validate specs - [SKILL.md](/kb/skills) — companion prose layer for agents - [x402](/kb/x402) — payment protocol that integrates via `x-payment-info` - [Agent Readiness](/agent-readiness) — how OpenAPI fits into the bigger picture --- ### llms.txt — LLM Context File > Plain-text file that explains your service to large language models. Source: https://agentgrade.com/kb/llms-txt ## What is llms.txt? llms.txt is a plain-text Markdown file served at the root of your domain (`/llms.txt`) that describes your service to large language models in human-readable prose. Where [OpenAPI](/kb/openapi) gives machines a structured spec and [robots.txt](/kb/robots-txt) tells crawlers what to fetch, llms.txt gives an LLM the *narrative context* it needs to understand what your service does, when an agent should use it, and which endpoints matter — in the format LLMs already process best. The spec was proposed by Jeremy Howard (fast.ai, Answer.AI) in September 2024. It is intentionally lightweight: no schema validation, no JSON parsing, no versioning. A Markdown file with an H1 heading and a one-line summary. The simplicity is the feature — anyone can author one in five minutes, and any LLM can read it without special tooling. ## Why llms.txt matters LLMs that browse the web have a triage problem. A single page on a modern site can be 200KB of HTML with most of the meaningful text behind JavaScript that the agent's fetcher can't render. Even when the agent can render JS, the signal-to-noise ratio is poor — navigation, ads, cookie banners, and tracking scripts dominate the bytes. llms.txt is the inverse: a single document, plain text, fits in a few KB, leads with what matters. An agent retrieving `https://example.com/llms.txt` gets: - What the service is, in one sentence - The endpoints or pages that matter, listed - Auth/payment rules in prose - Example calls the agent can copy This is the difference between "agent has to figure out your site" and "agent can decide whether your site is relevant in one fetch." For sites that want to be cited by ChatGPT, Claude, Perplexity, or any retrieval-augmented system, llms.txt is the cheapest possible affordance. ## How it works There is no protocol negotiation. Agents request `/llms.txt` the same way they request `/robots.txt`, expect `text/markdown` or `text/plain` back, and parse the Markdown. The file MUST start with an H1 heading; everything else is convention. A minimal valid llms.txt: ```markdown # Your Service Name > One-line description of what the service does. ## Overview What your service is for, in 2-3 sentences. Lead with the use case an agent would care about, not your company history. ## Endpoints - `GET /api/search?q=` — Search the catalog (free) - `POST /api/order` — Place an order (paid: x402) - `GET /api/status` — Health check (free) ## Authentication Free endpoints need no auth. Paid endpoints return HTTP 402 with an x402 payment challenge. Settle in USDC on Base. ## Examples Search for shoes: GET /api/search?q=shoes Place an order: POST /api/order {"sku": "ABC123", "qty": 1} ``` That's the whole spec. Headings beyond H1 are optional. Bulleted lists, fenced code blocks, and links work because they're Markdown — agents parse them with whatever Markdown library they already use. ## llms.txt vs other agent-readability files | File | Purpose | Format | Who reads it | |---|---|---|---| | `/llms.txt` | Narrative context for LLMs | Markdown | LLMs, browsing agents | | `/llms-full.txt` | Full text corpus, one file | Markdown / plaintext | RAG pipelines, embedded agents | | `/robots.txt` | Crawl permissions | Plaintext directives | Crawlers, bots | | `/sitemap.xml` | URL index for crawlers | XML | Search engines | | `/openapi.json` | Programmatic API spec | JSON/YAML | Code generators, API clients | | `/.well-known/mcp` | Tool-server protocol | JSON-RPC | MCP-capable agents | llms.txt does not replace any of these — it fills the gap between robots.txt (machine directives) and OpenAPI (machine specs) with a layer that is optimized for the *language* models that need to reason about what your site is for. ## The companion: llms-full.txt The llmstxt.org spec defines an optional sibling file: [llms-full.txt](/kb/llms-full-txt). Convention is the same — Markdown, served at the root — but the content is different. llms.txt is the *directory* (a brief overview with pointers). llms-full.txt is the *corpus* (the full concatenated text content of your site in a single file). Why both? Two different consumers: - **Browsing agents** with limited context windows want the directory — pick what to fetch next based on llms.txt. - **Non-browsing agents** (RAG pipelines, one-shot retrieval, embedded agents in mobile apps) want the corpus — ingest llms-full.txt once and answer questions without making more HTTP requests. Ship both if you can. llms.txt is the marketing front; llms-full.txt is the knowledge base. ## Who's publishing llms.txt Adoption has been broad across AI infrastructure, developer tools, and content platforms: - **Anthropic** publishes llms.txt at [docs.anthropic.com/llms.txt](https://docs.anthropic.com/llms.txt) listing every page in the developer docs. - **Cloudflare** publishes one at [developers.cloudflare.com/llms.txt](https://developers.cloudflare.com/llms.txt) for the Workers and platform docs. - **Stripe** publishes one for the API reference. - **Vercel, Mintlify, Netlify, Supabase, and Fly.io** all publish llms.txt for their developer documentation. - **Many open-source projects** ship llms.txt automatically via documentation generators (Mintlify and Docusaurus both have plugins). The pattern that's emerged: docs platforms publish llms.txt for their *documentation*, and product sites publish llms.txt for their *API surface*. Both are useful — agents discovering a product for the first time read the marketing llms.txt, then dive into the docs llms.txt when they need integration details. ## How to add llms.txt to your service ### 1. Decide what an agent needs to know Skip everything an agent doesn't need: your company story, design philosophy, marketing copy. Lead with what the service *does* and what endpoints exist. If you can't write the summary in one sentence, your service is too broad — split it. ### 2. Author the file Create `/llms.txt` as plain Markdown. Required: an H1 heading on the first line and a blockquote summary. Recommended: an "Endpoints" section, an "Authentication" section, and 2-3 example requests. Keep the whole file under 10KB. ### 3. Serve it at the root The file MUST be at `/llms.txt` — not `/.well-known/llms.txt`, not `/docs/llms.txt`. Agents probe the root directly. Most static-site hosts let you drop a file in `public/` or `static/` and serve it as-is. Set `Content-Type: text/markdown` or `text/plain`. Some agents reject `application/octet-stream`. ### 4. Link it from your homepage Add a `rel="alternate"` link in your HTML head so agents and crawlers discover it: ```html ``` ### 5. Ship llms-full.txt too Concatenate your documentation pages into a single Markdown file at `/llms-full.txt`. Most docs generators can emit this. If you're hand-rolling, a build step that walks your Markdown sources and joins them with H1 separators is enough. ## Common errors and debugging - **File served as HTML.** Static hosts sometimes wrap text files in their template. Check that `curl https://your-domain.com/llms.txt` returns the raw Markdown, not an HTML page that contains it. - **Missing H1 on first line.** The spec requires the document to start with `#`. Files that lead with a blockquote, frontmatter, or HTML comment fail parsers that expect the H1. - **Content negotiation overrides.** Some servers return HTML to browsers and Markdown to agents — but if the `Accept` header is wrong, agents get HTML. Always serve the file directly; don't gate it behind content negotiation. - **Stale llms.txt.** Authors ship the file once and never update it. Agents that re-fetch on a schedule see stale endpoints. Treat llms.txt as a build artifact — regenerate when the API changes. - **Too long.** Files over 50KB get truncated by some agents. If your llms.txt is that long, you actually want llms-full.txt — split into the directory and the corpus. agentgrade's scanner probes `/llms.txt` directly and checks that the file exists, starts with an H1, and has non-trivial content. ## Frequently asked questions ### Is llms.txt the same as robots.txt? No. robots.txt is directives for *crawlers* (what URLs they're allowed to fetch). llms.txt is *context* for LLMs (what your service is and how to use it). A site can and should publish both. ### Do search engines read llms.txt? Some do, informally. Google has not committed to anything official. The primary consumers are LLM-powered tools — ChatGPT browsing, Claude with web search, Perplexity, Cursor's @web — which retrieve llms.txt when summarizing or citing your service. ### Should I author llms.txt by hand or generate it? Hand-author the top-level llms.txt — it's short (10-50 lines) and the editorial judgment matters. Generate llms-full.txt from your docs build pipeline — it's long and mechanical. ### What if my service has paid endpoints? Describe the payment in prose ("Paid endpoints return HTTP 402 with an x402 payment challenge"). Don't try to encode payment metadata in llms.txt — that's what [x402](/kb/x402) and [OpenAPI](/kb/openapi) `x-payment-info` are for. ### Does llms.txt replace OpenAPI? No. OpenAPI is a strict machine spec — code generators consume it directly. llms.txt is narrative — LLMs read it to understand intent. Ship both: OpenAPI for tooling, llms.txt for context. ### How is llms.txt different from a SKILL.md? [SKILL.md](/kb/skills) is *instructional* — a playbook the agent follows to perform a task. llms.txt is *descriptive* — context about what the service is for. SKILLs go inside agent runtimes (Claude Code, Cursor, etc.) and tell an agent *how to operate*. llms.txt sits on your site and tells *any* LLM *what you are*. ### Does the file have to be Markdown? The spec says Markdown. In practice, plaintext works — most LLMs ignore Markdown syntax anyway. But Markdown is the recommended format because the headings give structure that agents can use to skip to the relevant section. ### Will publishing llms.txt help my SEO? Indirectly. Traditional search engines don't rank pages on llms.txt presence. But AI Overviews, Perplexity, ChatGPT search, and Bing's generative answers increasingly cite sources — and a clear, well-structured llms.txt makes you a more citable source. That's GEO (generative engine optimization), the AEO-adjacent practice that's emerged alongside LLM-powered search. ## Spec maturity **Community standard.** Defined at [llmstxt.org](https://llmstxt.org/) by Jeremy Howard, September 2024. No formal governance body, but widely adopted by major AI infrastructure providers (Anthropic, Cloudflare, Stripe, Vercel) and docs platforms (Mintlify, Docusaurus). The spec is short and stable — no breaking changes expected. ## Learn more - [llmstxt.org](https://llmstxt.org/) — Specification - [llms-full.txt](/kb/llms-full-txt) — The full-content companion - [Anthropic llms.txt](https://docs.anthropic.com/llms.txt) — Reference implementation - [Agent Readiness](/agent-readiness) — How llms.txt fits in the broader landscape ## Cross-references {#cross-references} If your site also serves OpenAPI, MCP, SKILL.md, x402, A2A, or WebMCP, mention them inside llms.txt. The point of llms.txt is to be the *one document* an agent reads to learn what your service offers — if reading it doesn't reveal that you have a paid API or an MCP endpoint, the agent has to probe well-known paths on the chance they exist. Most won't. A good cross-reference is a one-line bullet with the URL the agent can fetch: - `POST /api/order` — Place an order (paid via x402 — see `/.well-known/x402.json`) - MCP server: `/mcp` - Full corpus for RAG: `/llms-full.txt` - OpenAPI spec: `/openapi.json` - SKILL.md (agent playbook): `/skill.md` - A2A agent card: `/.well-known/agent.json` - WebMCP manifest: `/.well-known/webmcp.json` No special section is required — inline mentions count. AgentGrade's scan emits one optional sub-check per resource your site exposes: if the scanner detected your `/mcp` endpoint but `llms.txt` never says the word "mcp", that's a soft fail flagging the gap. These are optional checks: they don't reduce your overall score, but they surface where your llms.txt is *less useful than it could be* for the agents reading it. ## What counts as meaningful content {#meaningful-content} The llms.txt spec only requires an H1, but a file with just a title teaches an agent nothing. AgentGrade's *Meaningful content* check confirms the file has at least one of three structural signals that give an agent something to act on: 1. **A list of markdown links** — at least three bullets in `- [Title](url)` form. This is the docs-directory shape used by Anthropic, Stripe, Cloudflare, Vercel, Mintlify, and most platform llms.txt files. Each link points the agent at real content it can fetch and read. 2. **A fenced code block** — anything wrapped in triple-backticks. A `curl` example, a sample JSON response, a snippet showing how to authenticate. This is the API-surface shape: one file describes one service with a working example. 3. **A named operational section** — an H2 heading like `## Endpoints`, `## API`, `## Routes`, `## Authentication`, `## Examples`, `## Usage`, or `## Quick Start`. These names signal "this is operational content for agents" rather than marketing prose. Any **one** of the three is enough to pass. Most well-formed llms.txt files hit at least two — for example, agentgrade.com's own file has link bullets in the Knowledge Base section *and* a `## Example` section *and* a `curl` fence. ### What fails A marketing footer dressed up as an llms.txt: ```markdown # Acme > We make great products. ## Product - Home - About - Pricing ## Legal - Terms - Privacy ``` Fifteen lines, passes the old line-count check, but: no markdown links (the bullets are plain text labels), no code block, no operational section. An agent reading this learns Acme exists and has pages — but nothing it can navigate to. Fails the *Meaningful content* check. ### Why links matter llms.txt is a routing file, not a content file (that's what [llms-full.txt](/kb/llms-full-txt) is for). Its job is to point agents to the URLs they should fetch next. Plain-text bullets like `- About` give the agent a label with no URL — it can't follow the lead. A markdown link like `- [About](/about) — company background` gives it a URL plus a one-line description so it can decide whether the page is worth fetching. The simplest fix when this check fails: turn your plain-text bullets into markdown links, or add one fenced example showing an agent how to call your service. --- ### llms-full.txt — Full Content for LLMs > Companion to llms.txt: the entire site text concatenated into one file so non-browsing LLMs can ingest your content in a single fetch. Source: https://agentgrade.com/kb/llms-full-txt ## What is llms-full.txt? llms-full.txt is the companion file to `/llms.txt` — same [llmstxt.org](https://llmstxt.org/) convention, but instead of *listing* your important pages, it contains the **entire concatenated text content** of your site in a single file. One fetch, the whole corpus. ## When does it help? llms.txt is a directory. An agent reads it, picks a relevant URL, fetches that page, repeats. That works well for agents that can browse. llms-full.txt is the full text. An agent reads it once and has everything. That matters for: - **No-browse retrieval pipelines** — RAG systems that ingest text but cannot make follow-up HTTP requests - **Embedded agents** — running in environments without arbitrary network access - **One-shot context** — when an LLM needs the whole knowledge base in its context window up front - **Training and fine-tuning ingestion** — when content goes into a model build pipeline rather than a runtime fetch For browsing agents, llms-full.txt is harmless overhead (they can choose not to fetch it). For non-browsing agents, it is the difference between "the agent has your docs" and "the agent has nothing about you." ## Format Plain text or markdown. Standard convention is to start with the same header as llms.txt, then concatenate every page in your llms.txt directory, separated by clear headings or `---` separators. Example skeleton: ```markdown # Your Service Name > Brief one-line description. ## Overview Your Service does X, Y, Z. --- ### /docs/getting-started (full markdown content of the getting-started page) --- ### /docs/api-reference (full markdown content of the API reference) (...continues for every page) ``` ## How agents discover it Two ways: **1. `` in your HTML head** ```html ``` **2. Reference it from `/llms.txt`** Most llms.txt files end with an "Optional" section. Add the companion there so agents reading the directory know the full-content version exists. ```markdown ## Optional - [Full content](/llms-full.txt) — entire site text in one file ``` ## How to generate it Most modern docs platforms (Mintlify, Docusaurus with plugins, Hugo with templates) can auto-generate llms-full.txt from your source. If you maintain docs manually, a simple build step that concatenates your markdown source files works fine. **Regenerate when content changes.** A stale llms-full.txt is worse than none — it gives agents outdated information about your service. Wire generation into the same CI step that builds your docs. ## Tradeoffs - **Pro** — A single fetch gives consumers everything; no crawling needed - **Pro** — Strong AEO signal: ChatGPT, Perplexity, and Claude all weight machine-readable corpus availability - **Pro** — Works for agents without browsing capability (RAG pipelines, embedded agents) - **Con** — Larger response: tens of KB to several MB depending on docs size - **Con** — Must be regenerated when content changes - **Con** — Concatenates everything, including content you might prefer agents fetch piecemeal (paid docs, gated content) ## What agentgrade checks - **llms-full.txt found** — `/llms-full.txt` responds 200 with at least 20 characters of non-HTML content that does not look like a soft-404 - **llms-full.txt linked from HTML** (optional) — `` is present in your homepage `` The check is scored at weight 1 — lower than llms.txt itself at weight 2, reflecting that llms-full is recommended companion content rather than a primary directory file. ## Real-world examples - [docs.anthropic.com/llms-full.txt](https://docs.anthropic.com/llms-full.txt) — Anthropic's full Claude docs in one file - [agentgrade.com/llms-full.txt](https://agentgrade.com/llms-full.txt) — our own dogfood version, generated from this knowledge base - The original llmstxt.org spec describes both files as the canonical pair ## Spec maturity **Emerging standard, growing fast.** Defined alongside llms.txt at llmstxt.org. Adoption is rising, driven by AI-first docs platforms (Mintlify auto-generates it) and the spread of no-browse retrieval workflows. Sites that ship both files now position themselves as authoritative sources for AI answer engines. ## Learn more - [llmstxt.org](https://llmstxt.org/) — Specification - [llms.txt](/kb/llms-txt) — The companion directory file ## Related - [SKILL.md](/kb/skills) - [OpenAPI](/kb/openapi) --- ### agents.txt — Agent Access Policies > Defines what AI agents are allowed to do on your service. Source: https://agentgrade.com/kb/agents-txt ## What is agents.txt? agents.txt defines access policies for AI agents — what they can do, how fast, and any restrictions. Think of it as [robots.txt](/kb/robots-txt) evolved for autonomous agents that don't just crawl but take actions. ## Example Serve at `/agents.txt`: ``` [agent:*] Allow-Actions: read, search Rate-Limit: 60/minute [agent:claude] Allow-Actions: read, search, post, vote Rate-Limit: 120/minute [agent:gptbot] Allow-Actions: read Rate-Limit: 30/minute ``` ## Key concepts - **Agent sections**: Define policies per agent or for all agents (`agent:*`) - **Allow-Actions**: What actions agents can take (read, write, post, etc.) - **Rate-Limit**: How many requests per time period - **Agent-specific rules**: Different agents can have different permissions ## Spec maturity **Emerging standard — not yet formal.** Multiple proposals exist: - An [IETF Internet-Draft](https://www.ietf.org/archive/id/draft-srijal-agents-policy-00.html) defines a formal policy file format - Community proposals on GitHub define alternative formats The concept is gaining traction but the format is not finalized. The example above follows the most common community convention. ## Learn more - [IETF agents-policy draft](https://www.ietf.org/archive/id/draft-srijal-agents-policy-00.html) — Formal specification proposal ## Related - [llms.txt](/kb/llms-txt) - [SKILL.md](/kb/skills) - [OpenAPI](/kb/openapi) --- ### Organization JSON-LD — Entity Graph for Agents > Declare your site as a schema.org Organization with sameAs links to your other canonical surfaces. The linked-data signal LLMs use to consolidate scattered surfaces into one entity in their knowledge graph. Source: https://agentgrade.com/kb/organization-jsonld ## What is Organization JSON-LD? Organization JSON-LD is a ` ``` ## What is sameAs? `sameAs` is the linked-data declaration that says "the entity described here is the same entity as the one at these other URLs." It is how machines consolidate scattered surfaces (your GitHub, your npm package, your LinkedIn, your registry listings) into a single entity in their knowledge graph. Without `sameAs`, each surface looks like an unrelated orphan to a crawler. With it, your authority and identity signals consolidate. ## Why it matters for AI agents LLM-driven assistants (ChatGPT browsing, Claude, Perplexity, Gemini via Google's index) rely heavily on entity resolution before they answer. When a user asks "what is acme.example?", the model: 1. Looks up the domain in its entity index 2. Pulls in all the surfaces marked `sameAs` — your code, your social, your packages, your registry listings 3. Builds a unified picture: "this is the company, here is what they ship, here is who runs them" JSON-LD is the most agent-accessible structured data format — it survives HTML-to-text conversion that strips microdata attributes, and it is directly parseable JSON. Schema.org is the vocabulary every major model has trained on. ## What agentgrade checks - **Organization JSON-LD** (optional) — A JSON-LD block on the homepage declares `@type: Organization` - **sameAs entity links** (optional) — The Organization block includes a `sameAs` array with at least two valid HTTPS URLs Both checks are optional — they do not penalize a passing score if absent — but they are strongly recommended for any site that wants AI assistants to talk about it accurately. ## What to link in sameAs The point of `sameAs` is to list **authoritative profiles for the same entity**. Good candidates: - **Code**: your GitHub organization, your GitLab, your public repos - **Packages**: npm, PyPI, crates.io, Maven Central, RubyGems - **Identity**: X/Twitter, LinkedIn company page, Mastodon, Bluesky - **Registries**: Crunchbase, Wikidata, Wikipedia entry - **Agent surfaces**: your listing on the Model Context Protocol Registry, Smithery, the [x402 Bazaar](/kb/bazaar) Pick the surfaces that are public, canonical, and stable. Do not link to dashboards, login pages, or surfaces that gate the content. ## Where to put it In your homepage ``. You can have multiple JSON-LD blocks — one for `WebApplication` or `WebSite`, one for `Organization`. Most rich-results tools handle both. ## Spec maturity **Established standard.** schema.org `Organization` and `sameAs` are mature, widely adopted, and supported by every major search engine and AI agent. Google's Rich Results Test validates them, and Google's [Article on knowledge panels](https://developers.google.com/search/docs/appearance/structured-data/article) lists them as primary identity signals. ## Learn more - [schema.org/Organization](https://schema.org/Organization) - [schema.org/sameAs](https://schema.org/sameAs) - [Google: Logo structured data](https://developers.google.com/search/docs/appearance/structured-data/logo) ## Related - [SKILL.md](/kb/skills) - [llms.txt](/kb/llms-txt) - [OpenAPI](/kb/openapi) --- ### robots.txt for Agents > Agent-specific directives in robots.txt that signal your site welcomes AI crawlers. Source: https://agentgrade.com/kb/robots-txt ## What are agent-specific robots.txt directives? Standard robots.txt controls web crawlers. By adding User-agent directives for AI-specific crawlers, you signal that your site is aware of and welcomes AI agents. ## Example Add to your `/robots.txt`: ``` User-agent: GPTBot Allow: / User-agent: Claude-Web Allow: / User-agent: Anthropic-AI Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / ``` ## Known AI crawlers | User-agent | Operator | Purpose | |---|---|---| | GPTBot | OpenAI | Training and browsing | | Claude-Web | Anthropic | Web browsing | | Anthropic-AI | Anthropic | Training | | PerplexityBot | Perplexity | Search and answers | | Google-Extended | Google | AI training | | Applebot | Apple | Siri and AI features | ## Why it matters Many sites block AI crawlers by default. Explicitly allowing them signals that your content and APIs are designed for agent consumption. ## Spec maturity **Established convention.** robots.txt is a longstanding web standard (RFC 9309). AI-specific User-agent strings are defined by each AI company. ## Learn more - [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) — robots.txt specification ## Related - [llms.txt](/kb/llms-txt) --- ### Identity Protocols > Decentralized identity standards that let agents verify who owns a domain. Source: https://agentgrade.com/kb/identity ## What are identity protocols? Identity protocols let agents verify who is behind a domain or service. They answer: "Who are you, and how can I verify that?" ## WebFinger Discover information about a user or service from their domain. Served at `/.well-known/webfinger`: ```json { "subject": "acct:service@yourdomain.com", "links": [ { "rel": "self", "type": "application/activity+json", "href": "https://yourdomain.com/actor" } ] } ``` **Spec**: [RFC 7033](https://www.rfc-editor.org/rfc/rfc7033) ## DID Document Decentralized Identifiers — self-sovereign identity URIs. Served at `/.well-known/did.json`: ```json { "id": "did:web:yourdomain.com", "verificationMethod": [{ "id": "did:web:yourdomain.com#key-1", "type": "Ed25519VerificationKey2020", "publicKeyMultibase": "z6Mk..." }] } ``` **Spec**: [W3C DID Core](https://www.w3.org/TR/did-core/) ## Nostr NIP-05 Maps human-readable names to Nostr public keys. Served at `/.well-known/nostr.json`: ```json { "names": { "service": "npub1..." } } ``` **Spec**: [NIP-05](https://github.com/nostr-protocol/nips/blob/master/05.md) ## AT Protocol DID Links a domain to a Bluesky/AT Protocol identity. Served at `/.well-known/atproto-did`: ``` did:plc:abc123xyz ``` **Spec**: [AT Protocol Identity](https://atproto.com/specs/did) ## Apple App Links & Android Asset Links Mobile deep-linking protocols that prove a domain is associated with a mobile app. - Apple: `/.well-known/apple-app-site-association` - Android: `/.well-known/assetlinks.json` ## Spec maturity All identity protocols listed here have **formal specifications**. WebFinger and DID are W3C/IETF standards. Nostr and AT Protocol are ecosystem-specific but well-defined. ## Related - [A2A](/kb/a2a) - [SKILL.md](/kb/skills) --- ### Skill file (SKILL.md) > A markdown file with YAML frontmatter that tells agents how to use your service. Source: https://agentgrade.com/kb/skills ## What is SKILL.md? SKILL.md is a single Markdown file — typically served at `/skill.md` — that tells an AI agent how to use your service. The file opens with YAML frontmatter declaring two fields, `name` and `description`, then free-form Markdown that documents the endpoints, auth model, and request shapes the agent needs to act. The spec is published at [agentskills.io](https://agentskills.io/specification) and has been adopted as a cross-vendor convention by Anthropic's Claude Code, OpenAI Codex, Cursor, GitHub Copilot, and Microsoft's Agent Framework. A SKILL.md file is the smallest possible artifact that lets an agent go from "I found this site" to "I know how to call it." ## Why SKILL.md matters LLM agents face the same onboarding cost humans do: figuring out what an unfamiliar service is for and how to call it correctly. With humans you write a quickstart guide; with agents you write a SKILL.md. The frontmatter `name` gives the agent a stable identifier it can register internally; the `description` is what shows up in tool listings and the agent's planner; the body is the playbook. The alternative — an agent reading your full docs site, OpenAPI schema, and homepage to reconstruct the same information — is slow, expensive, and lossy. SKILL.md collapses the onboarding step to a single fetch and a few hundred tokens. For sites that want agents to *actually use them* (not just discover them), SKILL.md is the highest-leverage file you can publish. ## How SKILL.md works There is no protocol handshake. An agent requests `/skill.md`, parses the YAML frontmatter for `name` and `description`, validates the name against the spec's regex, and reads the Markdown body as instructions. A minimal valid SKILL.md: ```markdown --- name: scan-url description: Scan any URL for AI agent readiness and return a 0-100 score. Use when a user asks how agent-ready a site is or wants to compare protocol support across sites. --- # Agent Readiness Scan Use `GET /api/v1/scan?url={url}` for the agent-friendly JSON response. ## Endpoints - `GET /api/v1/scan?url={url}` — flat JSON for agents - `GET /api/scan?url={url}` — full HTML report (for humans) - `GET /api/badge?url={url}` — SVG badge ## Auth None. All endpoints are free. ## Example `curl https://agentgrade.com/api/v1/scan?url=https://example.com` ``` That is the entire surface. Agents that find this file know enough to call your service correctly on the first try. ## SKILL.md vs other agent-readability files | File | Who serves it | What it answers | Format | |---|---|---|---| | **SKILL.md** | The service | *How do I use this service?* | Markdown + frontmatter | | **[OpenAPI](/kb/openapi)** | The service | *What endpoints exist and what's the schema?* | JSON/YAML | | **[MCP](/kb/mcp) manifest** | The MCP server | *What tools does this server expose over JSON-RPC?* | JSON-RPC | | **`ai-plugin.json`** | The service | *Plugin metadata for ChatGPT plugin format* (legacy) | JSON | | **[llms.txt](/kb/llms-txt)** | The service | *What is this site, in prose?* | Markdown | SKILL.md is the "playbook" layer; OpenAPI is the "schema" layer. The two compose: SKILL.md tells the agent which endpoints to call and when; OpenAPI provides the formal parameter schema. Agents that find both are strictly better off than agents that find either alone. `ai-plugin.json` predates SKILL.md and was tied to the original ChatGPT plugin format, which OpenAI has since deprecated. SKILL.md is the modern, cross-vendor successor. ## Frontmatter validation The agentskills.io spec is strict about two fields: - **`name`** — 1 to 64 characters, lowercase alphanumeric with single hyphens. Must match `^[a-z0-9]+(-[a-z0-9]+)*$`. No leading, trailing, or consecutive hyphens. Good: `scan-url`, `book-flight`, `order-status`. Bad: `ScanUrl`, `-scan`, `scan--url`, `scan_url`. - **`description`** — 1 to 1024 characters. Lead with the verb the agent should perform ("Scan a URL...", "Book a flight from..."). The description shows up in tool pickers and planner reasoning, so write it for an agent making a snap decision. Agents and scanners that conform to the spec will reject files with malformed frontmatter — your file effectively becomes invisible to them. ## Who's adopted SKILL.md The spec moved from a single-vendor convention to an interoperable standard over 2025: - **Claude Code** ([docs.claude.com/en/docs/claude-code/skills](https://docs.claude.com/en/docs/claude-code/skills)) — Anthropic's coding agent loads SKILL.md files from local directories and uses them to scope tool access. - **OpenAI Codex** — Adopted the same format for agent skill definitions. - **Cursor** — Reads SKILL.md from project workspaces for inline agent guidance. - **GitHub Copilot** — Honors the same frontmatter convention for repository-level agent instructions. - **Microsoft Agent Framework** — Cross-vendor implementations follow the agentskills.io spec. Because the format is just Markdown with YAML frontmatter, adoption cost is near zero: every agent that already parses Markdown can read it; every site that already publishes docs can author it. ## How to add SKILL.md to your site 1. **Pick the name.** One verb-noun phrase, lowercase-hyphen, under 64 chars. Match what the agent should "do" with your service. `book-flight` is good; `flights` is too vague. 2. **Write the description.** Two sentences. Sentence 1: what the agent accomplishes. Sentence 2: when to use it (the trigger phrase or context). 3. **Write the body.** Endpoints with HTTP verbs, auth model, one copy-pasteable curl example. Keep it under a screen of text — terse beats exhaustive. 4. **Serve at `/skill.md`.** Static file. `Content-Type: text/markdown`. 5. **Link from llms.txt or `/.well-known/`.** Optional but recommended — gives agents that find your llms.txt first a path to the skill file. A site with a valid SKILL.md and an [OpenAPI](/kb/openapi) spec is well-equipped for any agent that arrives knowing nothing. ## Common errors scanners flag - **Missing frontmatter delimiters** — the file does not start with `---`, so the YAML block is not detected. - **Invalid `name`** — uppercase, underscores, or leading/trailing hyphens. The most common failure. - **Description too short** — under 1 character (empty) or so short the agent has no signal. Aim for 80-200 chars. - **Description too long** — over 1024 characters. Move detail into the body. - **Wrong path** — served at `/skills.md` (plural), `/.well-known/skill.md`, or `/skill/index.md`. The spec is `/skill.md` at the document root. - **Wrong content-type** — served as `text/html` because a CMS auto-renders Markdown. Serve raw `text/markdown` or `text/plain`. ## FAQ **Is SKILL.md the same as Anthropic's "Claude Skills"?** The on-disk format is the same. "Claude Skills" is Anthropic's product name for the feature that consumes SKILL.md files; the file format itself is the cross-vendor agentskills.io spec. **Do I need both SKILL.md and OpenAPI?** Ship both if you have an API. SKILL.md answers "how do I use this?" in prose; OpenAPI answers "what is the exact schema?" formally. Agents use OpenAPI to validate requests and SKILL.md to decide whether to call the service at all. **What replaced `/skills.json`?** `/skills.json` was an early, hand-rolled manifest format that predated any standard. It has been superseded by SKILL.md for the "how to use" layer and OpenAPI's `x-payment-info` (see [OpenAPI](/kb/openapi)) for declaring paid endpoints. AgentGrade no longer scores `/skills.json`. **Where should the SKILL.md file live?** `/skill.md` at the document root. Some agents will also probe `/.well-known/skill.md` as a fallback — publishing both is harmless. **Can a site have multiple SKILL.md files?** The convention is one `/skill.md` per origin. Services with multiple distinct surfaces (search vs checkout vs support) typically describe each as a section within one file, or split into sub-services on subdomains. **Does SKILL.md replace [MCP](/kb/mcp)?** No — they answer different questions. SKILL.md tells an agent how to use your HTTP service via existing endpoints. MCP defines a JSON-RPC interface where the agent calls your server as a tool over a persistent connection. A site can publish both. **Does AgentGrade penalize sites that publish SKILL.md without OpenAPI?** No. SKILL.md is a positive signal on its own. OpenAPI is a separate check. **Will agents actually read this file?** Claude Code, Codex, Cursor, and Copilot read SKILL.md files in their workspaces today. Browsing agents that discover your site for the first time are increasingly probing `/skill.md` as part of the discovery handshake, alongside `/llms.txt` and `/openapi.json`. ## Spec maturity **Cross-vendor standard.** Defined at [agentskills.io](https://agentskills.io/specification). Adopted by Claude Code, OpenAI Codex, Cursor, GitHub Copilot, and Microsoft Agent Framework. Active development; the canonical implementation is well-documented. ## Learn more - [agentskills.io](https://agentskills.io/specification) — the specification - [Claude Code skills documentation](https://docs.claude.com/en/docs/claude-code/skills) — Anthropic's adoption - [OpenAPI](/kb/openapi) — companion schema layer - [MCP](/kb/mcp) — alternative for JSON-RPC tool servers - [Agent Readiness](/agent-readiness) — how SKILL.md fits into the broader picture --- ### Content Negotiation for AI Agents > How to serve agent-optimized responses based on User-Agent and Accept headers. Source: https://agentgrade.com/kb/content-negotiation ## What is content negotiation? Content negotiation means serving different responses based on what the client asks for. For AI agents, this means returning structured text (JSON, markdown, or plain text) instead of heavy HTML when the request comes from an agent. ## Why it matters AI agents strip HTTP headers before the model sees content. The model never sees Content-Type, status codes, or redirect chains — it only sees the body text. Serving HTML forces agents to parse DOM structure, extract text, and discard layout — wasting tokens and losing meaning. ## What we check **Agent UA gets non-HTML** — We send a request with a known AI agent User-Agent (e.g. `ClaudeBot`) and check if the response Content-Type is something other than `text/html`. Sites that detect agent UAs and serve `text/markdown`, `text/plain`, or `application/json` pass this check. **Accept: JSON returns JSON** — We send `Accept: application/json` and check if the response is valid JSON. This lets programmatic agents access structured data directly. **Accept: text returns text** — We send `Accept: text/plain` and check if the response is plain text or markdown. This is the simplest format for language models to consume. **Accept: markdown returns markdown** — We send `Accept: text/markdown` and check if the response is markdown content (Content-Type `text/markdown` or a markdown body). This is the format agents prefer when they want structure (headings, lists, links) without HTML weight. ## How to implement Detect agent User-Agents and `Accept` headers in your server middleware: ```javascript app.get('/', (req, res) => { const ua = (req.get('user-agent') || '').toLowerCase(); const accept = req.get('accept') || ''; const isAgent = /claude|gpt|anthropic|perplexity|gptbot/i.test(ua); // Markdown takes precedence when explicitly requested if (accept.includes('text/markdown')) { return res.type('text/markdown').sendFile('[llms.txt](/kb/llms-txt)'); } if (accept.includes('application/json')) { return res.json({ name: 'My Service', api: '/openapi.json' }); } if (isAgent || accept.includes('text/plain')) { return res.type('text/plain').sendFile('llms.txt'); } res.sendFile('index.html'); }); ``` ### Markdown specifics `Accept: text/markdown` is a newer convention (popularized by the llms.txt ecosystem) and not yet universally implemented. Returning markdown when requested means: - The body is valid markdown (headings, lists, links — no HTML tags) - The Content-Type header is `text/markdown` (or `text/plain` with markdown body) - The same URL serving HTML to a browser can serve markdown to an agent — the path doesn't change, only the representation For documentation pages, blog posts, and content pages, you typically already have a markdown source. Serve that source directly when `Accept: text/markdown` is sent, instead of converting to HTML first. ## Spec maturity **Established.** Content negotiation is defined in HTTP itself ([RFC 9110](https://www.rfc-editor.org/rfc/rfc9110)). The convention of returning agent-friendly formats based on User-Agent or Accept is widely used by AI-aware sites. ## Learn more - [RFC 9110 §12: Content Negotiation](https://www.rfc-editor.org/rfc/rfc9110#name-content-negotiation) - [llmstxt.org](https://llmstxt.org/) — Markdown-for-LLMs convention ## Related - [OpenAPI](/kb/openapi) --- ### Web Bot Auth (RFC 9421) > How AI agents prove their identity with cryptographic HTTP message signatures. Source: https://agentgrade.com/kb/web-bot-auth ## What is web-bot-auth? Web-bot-auth is an emerging standard for AI agents to prove their identity using cryptographic HTTP message signatures (RFC 9421). Instead of relying on User-Agent strings — which can be faked — agents sign their requests with a private key. Sites verify the signature against a published public key. ## Why it matters User-Agent detection is no longer reliable: - **ChatGPT Agent Mode** uses a fake Chrome UA (`Chrome/138.0.0.0`) — indistinguishable from a browser by UA alone - **Perplexity** runs stealth crawlers (3-6M requests/day) with generic Chrome UAs - **Google-Agent** uses standard Chrome UA strings The industry is moving to cryptographic identity. OpenAI, Google, Cloudflare, Shopify, Vercel, and Visa are backing this approach. IETF standardization is in progress. ## What AgentGrade checks **Signatures directory published** — We check for `/.well-known/http-message-signatures-directory`. This is the standard location where a site publishes its agent identity and the public keys that can be used to verify its signed requests. **Members declared** — The directory should contain a `members` array listing the agent identities (e.g., "ChatGPT", "GoogleAgent") that the site recognizes or acts as. **Public keys available** — Each member should have a `publicKeyUrl` pointing to a fetchable public key for signature verification. ## How to implement Publish a JSON file at `/.well-known/http-message-signatures-directory`: ```json { "members": [ { "name": "my-agent", "publicKeyUrl": "https://example.com/.well-known/keys/agent.pub" } ] } ``` To verify incoming signed requests from known agents (like ChatGPT), use the `web-bot-auth` npm package: ```bash npm install web-bot-auth ``` ## Known agent directories | Agent | Directory URL | |-------|---------------| | ChatGPT | `https://chatgpt.com/.well-known/http-message-signatures-directory` | ## Learn more - [RFC 9421: HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421) - [web-bot-auth npm package](https://www.npmjs.com/package/web-bot-auth) - [Cloudflare blog: web-bot-auth](https://blog.cloudflare.com/web-bot-auth/) - [SeatGeek implementation](https://chairnerd.seatgeek.com/chasing-signature/) ## Related - [A2A](/kb/a2a) - [SKILL.md](/kb/skills) --- ### A2A — Agent-to-Agent Protocol > Google's protocol for AI agents to discover and communicate with other agents. Source: https://agentgrade.com/kb/a2a ## What is A2A? A2A (Agent-to-Agent) is an open protocol for AI agents to discover, authenticate with, and communicate with each other. A site that hosts an agent publishes an **Agent Card** — a JSON manifest at `/.well-known/agent.json` — that declares the agent's identity, capabilities, skills, and the URL where other agents can reach it. The protocol was introduced by Google in April 2025 and donated to the Linux Foundation later that year as part of an industry push for an open agent interoperability layer. A2A sits in a different layer than [MCP](/kb/mcp). MCP defines how an agent calls a *tool*; A2A defines how an agent calls *another agent*. The two compose: an MCP server exposes tools to a single agent; an A2A endpoint exposes a whole agent to the wider agent network. ## Why A2A matters The agent ecosystem in 2025-2026 is converging on a multi-agent pattern: a user-facing agent (Claude, ChatGPT, Gemini) coordinates with specialist agents hosted by other services. The booking agent on a travel site is one specialist; the refund agent on an e-commerce site is another. The user-facing agent needs a way to discover those specialists, learn what they can do, and call them — without per-vendor SDK integration. A2A is the protocol for that discovery and routing layer. An Agent Card tells the calling agent: - **Who you are** — name, description, version, vendor. - **What you can do** — a list of named skills, each with its own description. - **How to reach you** — the URL of your agent endpoint and supported transport (HTTP, SSE, WebSocket). - **How to authenticate** — the auth scheme, including OAuth flows or API key handoff. Without A2A, an agent that finds your site has no path from "this site exists" to "this site can help me with refunds via this exact endpoint and this exact auth." A2A is the bridge. ## How A2A works There are two pieces: the **Agent Card** (discovery) and the **A2A endpoint** (execution). **1. The Agent Card** is a static JSON file at `/.well-known/agent.json`. Other agents fetch it the same way they fetch any well-known file. A minimal valid Agent Card: ```json { "name": "Acme Support Agent", "description": "Handles customer inquiries, refunds, and order status.", "version": "1.0.0", "url": "https://acme.example.com/agent", "capabilities": { "streaming": true, "pushNotifications": false }, "skills": [ { "id": "process-refund", "name": "Process Refund", "description": "Issue a refund for a given order ID." }, { "id": "order-status", "name": "Order Status", "description": "Look up the status of an order by order ID." } ] } ``` **2. The A2A endpoint** at the URL declared in the card accepts requests for those skills. The protocol uses JSON-RPC-style messaging over HTTP (with optional SSE for streaming) and supports a structured message format with text, file, and structured-data parts. The calling agent reads the card, picks a matching skill, authenticates as needed, and POSTs a message to the endpoint. The receiving agent processes it and responds — possibly with streaming if the card declared streaming support. ## A2A vs other agent-interop protocols | Protocol | Purpose | Transport | What it exposes | |---|---|---|---| | **A2A** | Agent calls another agent | HTTP/SSE | A whole agent with named skills | | **[MCP](/kb/mcp)** | Agent calls a tool | JSON-RPC over HTTP/SSE | Discrete tools on a server | | **[OpenAPI](/kb/openapi)** | Agent calls a REST API | HTTP | API endpoints with schemas | | **[SKILL.md](/kb/skills)** | Tells agent how to use a service | Static Markdown | Prose playbook for an HTTP service | | **[WebMCP](/kb/webmcp)** | Agent calls a website through the browser | Browser API | Annotated HTML forms | A2A is the only one of these where the *callee is itself an agent*, not a tool or a static API. The difference matters: A2A endpoints can reason, plan, and delegate further. They're peers, not functions. ## Agent Card fields The full Agent Card spec includes: - **`name`** — human-readable agent name. Required. - **`description`** — what the agent does, in prose. Required. - **`url`** — where the A2A endpoint lives. Required. - **`version`** — agent version string. Recommended. - **`provider`** — vendor info: organization name, URL. - **`capabilities`** — feature flags: `streaming`, `pushNotifications`, `stateTransitionHistory`. - **`skills`** — array of skill objects with `id`, `name`, `description`, and optional `inputModes` / `outputModes`. - **`authentication`** — auth scheme details: type (`bearer`, `oauth2`, `apiKey`), and scheme-specific fields. - **`defaultInputModes`** / **`defaultOutputModes`** — supported content types for skill calls (`text`, `file`, `data`). AgentGrade's A2A check validates that the card exists, parses as JSON, and contains at least `name` and `url`. Richer cards score the same on the binary check but signal to consuming agents that the service is well-integrated. ## Who's using A2A A2A launched with backing from Google and a coalition of partners. Public adopters and integrators in 2025-2026 include: - **Google** — the original author; A2A is integrated into Vertex AI Agent Builder. - **Linux Foundation** — A2A was donated to the LF to ensure neutral governance. - **Atlassian** — published Agent Cards for Jira and Confluence agents. - **Box, MongoDB, PayPal, SAP, ServiceNow, Workday** — early ecosystem partners listed at launch. The adoption pattern mirrors OpenAPI's early years: a Google-led specification adopted by enterprise SaaS first, with broader uptake following as multi-agent orchestration moves from research demos to production. ## How to add A2A to your service 1. **Decide if you have an agent to expose.** A2A is for *agents*, not static APIs. If your service is a REST API, ship [OpenAPI](/kb/openapi); if it's an MCP tool server, ship [MCP](/kb/mcp). If you have a service that *plans, reasons, or holds conversation state*, A2A is the right layer. 2. **Author the Agent Card** at `/.well-known/agent.json`. Start with the minimal fields (`name`, `description`, `url`, `skills`). Add `capabilities` and `authentication` as you implement them. 3. **Stand up the A2A endpoint** at the URL in the card. The Python SDK at [github.com/google/A2A](https://github.com/google/A2A) is the reference implementation; JavaScript and other ports exist. 4. **Implement at least one skill.** Don't ship an Agent Card listing skills you haven't built — agents that call you and get errors will deprioritize you. 5. **Test with a caller.** Use a generic A2A client to verify the card parses and the endpoint responds to a real message. 6. **Link from llms.txt or SKILL.md** so non-A2A-native discovery paths find your card. ## Common errors scanners flag - **No Agent Card** — `/.well-known/agent.json` returns 404. AgentGrade's A2A check is informational; this is not a score penalty, but a missed opportunity for services that *do* host an agent. - **Card returns HTML, not JSON** — Single-Page App catch-all routes return the homepage HTML for every path. Serve raw JSON from a static file or framework route that bypasses the SPA fallback. - **Missing `name` or `url`** — the spec's minimum required fields. Cards without these are not callable. - **`url` points to a 404** — the card declares an endpoint that doesn't exist. - **Skills array empty or missing** — the card validates but there's nothing for an agent to call. - **CORS blocks the fetch** — agents that read the card from a browser context need `Access-Control-Allow-Origin: *` or a permissive policy. ## FAQ **Is A2A a replacement for [MCP](/kb/mcp)?** No. They solve different problems. MCP is for an agent calling a tool; A2A is for an agent calling another agent. A site that hosts a tool server publishes MCP; a site that hosts a whole agent publishes A2A. Some sites do both. **Do I need A2A if I have [OpenAPI](/kb/openapi)?** Only if your service is an agent (reasons, plans, holds state across turns). REST APIs ship OpenAPI; agents ship A2A. The two coexist — an A2A agent often calls REST APIs documented in OpenAPI as part of its implementation. **Is A2A required for AgentGrade scoring?** A2A is an informational check, not a scored one. AgentGrade flags the presence of an Agent Card but does not penalize its absence for sites that aren't agents. Static-content sites should not publish empty Agent Cards. **What happens if my Agent Card lies about capabilities?** The card declares what the agent claims; the calling agent will eventually find out via failed requests. As with any declared-vs-verified signal, lying degrades trust over time. AgentGrade's check is currently presence-only and does not probe declared skills. **Can I publish multiple Agent Cards?** The spec is one card per origin. Services that host multiple distinct agents typically split them onto subdomains. **What's the relationship between A2A and Google's Vertex AI Agent Builder?** Vertex AI Agent Builder is Google's hosted agent runtime; it produces and consumes A2A. A2A itself is the open protocol, donated to the Linux Foundation, that Vertex and any other agent runtime can speak. **How do agents discover Agent Cards?** The same way they discover any well-known file: a direct fetch of `/.well-known/agent.json`, or following a link from `/llms.txt`. Agent indexes and directories are starting to crawl Agent Cards the way search engines crawl sitemaps. **Does A2A support payment?** The spec includes capability declarations but payment is delegated to the underlying auth scheme. Sites that combine A2A with [x402](/kb/x402) typically declare the payment requirement in the skill description and gate the endpoint with a 402 challenge. ## Spec maturity **Open standard, early adoption.** A2A is governed by the Linux Foundation. The Python SDK is the reference implementation. Early adopters are enterprise SaaS; broader adoption is following as multi-agent orchestration moves into production. ## Learn more - [A2A Protocol Specification](https://google.github.io/A2A/) — official docs - [A2A GitHub repository](https://github.com/google/A2A) — reference implementations - [MCP](/kb/mcp) — companion protocol for tool calls - [Agent Readiness](/agent-readiness) — how A2A fits into the broader picture --- ### WebMCP — Browser Agent Interaction > A W3C draft protocol that lets browser-based AI agents interact with websites through annotated forms. Source: https://agentgrade.com/kb/webmcp ## What is WebMCP? WebMCP is a W3C Community Group draft proposal that lets AI agents interact with websites *through the user's browser* rather than over a separate API. Sites annotate existing HTML forms with attributes that declare them as callable tools; the browser exposes those tools to an agent via a new `navigator.modelContext` API. The proposal is backed by Google and Microsoft and is currently available behind an experimental flag in Chrome Canary. The name is a deliberate echo of [MCP](/kb/mcp), Anthropic's Model Context Protocol, but the two are distinct specs that solve different problems. MCP defines a JSON-RPC protocol for an agent to call tools on a server. WebMCP defines a browser API for an agent to call tools that already exist as forms on a page. They share the "tools" concept and not much else. ## Why WebMCP matters The traditional path for agent-website integration is: the site publishes an API ([OpenAPI](/kb/openapi)) or an MCP server, the agent authenticates separately, and calls run over a backchannel. This works for first-party agent integrations but breaks down for the long tail of websites that have valuable functionality but no API team to build a parallel surface. WebMCP takes the inverse approach: the site already has a form to do the thing, the user is already logged in, the browser already has the session. Why build a parallel API? Annotate the existing form with a few attributes, and an agent in the browser can call it directly — inheriting the user's auth, the site's CSRF protections, and the browser's permissions model for free. The use cases are mostly *user-in-the-loop* agent flows: an agent acting on behalf of a logged-in user on the site the user is currently visiting. Booking a flight, filing a support ticket, posting a comment — anywhere a form would normally take a human submission. ## How WebMCP differs from [MCP](/kb/mcp) | | MCP | WebMCP | |---|---|---| | Who calls whom | Agent → server | Agent → browser → server | | Where the agent runs | Outside the browser | Inside or alongside the browser | | Authentication | OAuth 2.1 / API keys | The user's existing browser session | | Transport | JSON-RPC over HTTP/SSE | Browser internal API (`navigator.modelContext`) | | What you build on the site | A dedicated MCP endpoint | Attribute annotations on existing HTML forms | | Connection lifetime | Persistent across sessions | Ends when the tab closes | | Discovery | MCP registry / well-known | DOM scan + `/.well-known/webmcp.json` manifest | The architectures are nearly opposites. MCP gives external agents a stable backend interface; WebMCP gives in-browser agents access to the existing frontend. A site can publish both: MCP for headless agent integrations, WebMCP for user-in-the-loop browser flows. ## How WebMCP works There are two cooperating pieces: **DOM annotations** (per-page) and an optional **manifest** (per-site). **1. Annotated forms.** A site marks any HTML form as agent-callable with three custom attributes: ```html
``` The browser-side WebMCP runtime scans the DOM for `tool-name` elements, builds a tool list, and exposes it to the agent. When the agent invokes `search-products` with `q: "running shoes"`, the browser fills in the form and submits it on the agent's behalf — through the user's session. **2. The manifest.** A site can publish `/.well-known/webmcp.json` declaring tools that may not be visible on every page (or that involve multi-step flows). The manifest format mirrors MCP's tool descriptors: ```json { "spec": "webmcp/0.1", "tools": [ { "name": "search-products", "description": "Search the product catalog", "url": "/search", "method": "GET", "parameters": [ { "name": "q", "type": "string", "description": "Search query" } ] } ] } ``` The manifest gives agents a discovery path that doesn't depend on DOM scraping. AgentGrade checks both: a present `/.well-known/webmcp.json` and any `tool-name` attributes on the homepage. ## WebMCP vs other agent-interop layers | Spec | Where the agent lives | What's exposed | Auth | |---|---|---|---| | **WebMCP** | In the browser | Annotated HTML forms | User's session | | **[MCP](/kb/mcp)** | Outside the browser | JSON-RPC tools on a server | OAuth / API keys | | **[A2A](/kb/a2a)** | Anywhere | A whole agent with named skills | Agent-level auth | | **[OpenAPI](/kb/openapi)** | Anywhere | REST endpoints with schemas | Per-endpoint scheme | | **[SKILL.md](/kb/skills)** | Anywhere | Prose playbook | N/A | WebMCP is the only one that runs inside the user's browser. The others all assume the agent is on a server or in a chat client; WebMCP assumes the agent is the *user's user agent*, with all the trust and authority that implies. ## Current status WebMCP is a **Draft Community Group Report** under the W3C Web Machine Learning Community Group. As of mid-2026: - The spec is published at [webmachinelearning.github.io/webmcp](https://webmachinelearning.github.io/webmcp/). - The reference implementation is behind an experimental flag in Chrome Canary (chrome://flags/#enable-web-mcp). - It is **not** shipped in any production browser. Firefox and Safari have not announced plans. - The spec is expected to change before reaching W3C Working Draft status. Because the user-side surface is gated on browser support, sites that ship WebMCP today are early — they're betting on the spec stabilizing and browsers shipping. The cost of being early is low (a few HTML attributes); the upside is being agent-callable for the first wave of in-browser agents that adopt the spec. ## Who's backing WebMCP - **Google** — Chrome team published the proposal and ships the prototype. - **Microsoft** — co-editor on the spec; co-publisher of a [Chrome developer blog post](https://developer.chrome.com/blog/webmcp-mcp-usage) explaining the relationship between MCP and WebMCP. - **W3C Web Machine Learning Community Group** — the governance body. Notably absent: Anthropic (the MCP author) and OpenAI. The spec is positioned as complementary to MCP, not competitive, but the overlap in naming has caused some confusion. ## How to add WebMCP to your site 1. **Pick the forms that matter.** Don't annotate every form on the site. Pick the 1-3 most valuable agent-callable actions (search, submit, subscribe). 2. **Add the attributes.** `tool-name` on the `
` element, `tool-description` on the form, `tool-param-description` on each ``. 3. **Use lowercase-hyphen names.** `search-products`, not `searchProducts`. 4. **Write descriptions for an agent, not a human.** Lead with the verb. Be specific about the parameter type (e.g., "Search query string, 1-200 chars" not just "search"). 5. **Publish the manifest.** `/.well-known/webmcp.json` listing the same tools. This is the discovery path for agents that don't scan the DOM. 6. **Test in Chrome Canary** with the experimental flag enabled. ## Common errors scanners flag - **No manifest** — `/.well-known/webmcp.json` returns 404. The check is informational; site can still expose form-level tools. - **Manifest returns HTML** — SPA catch-all routes return the homepage instead of the JSON file. - **Form annotations without descriptions** — `tool-name` is set but `tool-description` is missing, so agents can't tell what the form does. - **Invalid tool name** — uppercase, underscores, or special chars in `tool-name`. - **Annotations on non-form elements** — only `` elements should carry `tool-name`. ## FAQ **Is WebMCP a fork of [MCP](/kb/mcp)?** No. They are separate specs from separate working groups with separate architectures. They share the "tools" concept (a named, parameterized action an agent can call) but the protocols, transports, and trust models are completely different. The naming is unfortunate. **Will Chrome's WebMCP work in Firefox or Safari?** Not currently. Firefox and Safari have not implemented the spec. Cross-browser support depends on the spec maturing past Draft status, which is a multi-year process. **Should I add WebMCP if I already have MCP?** Yes if your service has a website with forms that users fill out today. The two address different agent contexts: server-side agents use MCP; user-in-the-loop browser agents use WebMCP. Sites with both let agents pick the right surface. **Does AgentGrade penalize sites without WebMCP?** No. WebMCP is an informational check. The score is not penalized for missing it. It is a positive signal when present. **Is WebMCP secure?** The security model inherits from the browser: same-origin enforcement, the user's existing session, CSRF protection. An agent can only call tools the user could call manually on the same page. There is no escalation surface beyond what the user already has. The risk is that the agent acts on the user's behalf without the user understanding what's happening — a UX problem, not a security boundary problem. **What replaces `/.well-known/webmcp.json` if I host my forms in an iframe?** Same-origin rules apply: an agent in the parent page can call form-level tools in same-origin iframes. Cross-origin iframes are isolated. The manifest should be on the iframe's origin. **Does WebMCP work with single-page apps?** Yes, but forms must be present in the DOM at the time the agent scans. SPAs that hydrate forms on user interaction may need to declare them in the manifest instead. **When will WebMCP ship to stable Chrome?** No public timeline. The spec is Draft; production rollout depends on spec stability, security review, and demonstrated developer adoption. ## Spec maturity **Draft, not shipped.** WebMCP is a W3C Community Group draft, available behind an experimental flag in Chrome Canary only. The spec may change. Adoption is early — annotating forms is cheap insurance for sites that want to be ready when production browser support arrives. ## Learn more - [WebMCP specification](https://webmachinelearning.github.io/webmcp/) — the W3C draft - [WebMCP GitHub repository](https://github.com/webmachinelearning/webmcp) — proposals and discussions - [Chrome: When to use WebMCP and MCP](https://developer.chrome.com/blog/webmcp-mcp-usage) — the official positioning - [MCP](/kb/mcp) — the server-side counterpart - [A2A](/kb/a2a) — agent-to-agent protocol --- ### Infrastructure — Cache, Errors, Redirects > HTTP-level behaviors agents rely on: cache headers, structured errors, HTTPS redirect, sitemap. Source: https://agentgrade.com/kb/infrastructure ## What is "Infrastructure"? The Infrastructure group covers low-level HTTP behavior that determines whether agents can interact with your origin reliably. None of these are protocol-specific — they apply to any site, whether it serves an [MCP](/kb/mcp) endpoint, an [OpenAPI](/kb/openapi) spec, or just static content. ## HTTPS redirect If a human types `yourdomain.com` into a browser, the browser tries HTTPS first and most of the time gets there silently. Agents don't do that. They follow whatever URL they were handed — and if that URL is `http://yourdomain.com` and your server doesn't redirect, the agent's HTTP library either refuses to send the request (most modern AI clients block plain HTTP) or sends it unencrypted (older ones). Either way, the agent never reaches your real site. **Fix**: respond with `301` or `308` from `http://yourdomain.com/*` to the same path on HTTPS. Most platforms do this for you (Cloudflare, Vercel, Heroku, build.io). If you self-host, add an nginx `server` block listening on port 80 that issues the redirect. ## Sitemap.xml Agents discover your URL surface from `/sitemap.xml`. Without it, the only paths an agent can reach are those linked from your homepage or declared in [llms.txt](/kb/llms-txt) — typically a small fraction of your real content. **Fix**: serve an XML sitemap at `/sitemap.xml`. Frameworks like Next.js, Hugo, Jekyll, and Wordpress generate one automatically. For a hand-rolled sitemap: ```xml https://yourdomain.com/ https://yourdomain.com/docs ``` ## Cache headers Every well-configured server emits at least one of `Cache-Control`, `ETag`, or `Last-Modified` on responses. Without them, agents have no way to know whether to re-fetch or use a cached copy — they either re-download every time (slow, expensive) or stale-cache forever (incorrect data). **Fix**: most web frameworks set these by default for static files. For dynamic responses, set `Cache-Control` explicitly: ```javascript // Express — for content that changes occasionally res.set('Cache-Control', 'public, max-age=3600'); // For never-cache responses (still satisfies the check): res.set('Cache-Control', 'no-store'); ``` `no-store` counts as a valid cache header — it tells the agent "do not cache this," which is itself useful information. The check fails when *no* cache header is present at all. ## Structured errors (JSON 404) When an agent hits a wrong URL, an HTML 404 page (with nav, footer, search box, etc.) wastes tokens and can fool a confused agent into thinking it got real content. A JSON 404 returns a small, parseable error: ```json { "error": "not_found", "path": "/wrong-url" } ``` **Fix**: use content negotiation in your 404 handler. Return JSON when the client sends `Accept: application/json`, HTML otherwise. Express example: ```javascript app.use((req, res) => { res.status(404); res.format({ 'application/json': () => res.json({ error: 'not_found', path: req.path }), 'text/html': () => res.send('

404 Not Found

'), default: () => res.json({ error: 'not_found', path: req.path }), }); }); ``` This change is invisible to humans (browsers send `Accept: text/html` and get HTML) but transformative for agents (they send `Accept: application/json` and get JSON they can parse). ## Rate limit headers (optional) If your service rate-limits, expose the state via `X-RateLimit-Remaining`, `X-RateLimit-Reset`, or `Retry-After`. Agents that hit a limit can wait the right amount of time and retry instead of failing. This check is optional because most sites don't rate-limit at all — only API providers and SaaS apps do. Static sites have nothing to expose. ## Why these are required None of these break a website for humans. A site missing cache headers still loads in a browser; a site that returns HTML 404s still shows a friendly error page; a site with no sitemap still ranks in Google. That's exactly why they're easy to overlook — humans don't feel the problem. Agents do. Every one of these is the difference between "the agent gets the data" and "the agent gets noise, wastes tokens, or gives up." Marking them required is how we keep them from being silently skipped. ## Related - [WebMCP](/kb/webmcp) --- ### JavaScript-Rendered Sites and Agent Visibility > Why client-rendered SPAs are invisible to AI crawlers, what it does and does not break, and how to fix it. Source: https://agentgrade.com/kb/javascript-rendering ## What is "JavaScript-rendered"? A page is JavaScript-rendered when the HTML the server sends is essentially empty — the actual content (headings, prose, product information) only appears after the browser downloads and runs a JavaScript bundle that mounts a framework like React, Vue, Svelte, or Angular into an empty root element. A typical JS-rendered homepage looks like this on the wire: ```html
``` The visible `` content is zero bytes. Everything you see in a browser was constructed client-side after the JS ran. ## Why this matters for agents Most AI crawlers do not execute JavaScript. When ChatGPT, Claude, Perplexity, Gemini fetches, or a typical agent framework hits your URL, they get the raw HTML — and your homepage is blank. Your product description, headings, links, and prose are invisible to them. This is a real harm for retrieval and summarization. An LLM asked "what does this company do?" while citing your URL will produce a poor answer or hallucinate, because there is nothing on the page to read. ## What this does *not* break Tags emitted into the static `` of your HTML are unaffected by client-side rendering. Even a fully JS-rendered SPA can pass these checks if the build tool populates the head correctly: - JSON-LD structured data (`