Content Negotiation for AI Agents

What is content negotiation?

Content negotiation means serving different responses based on what the client asks for. For AI agents, this means returning structured text (JSON, markdown, or plain text) instead of heavy HTML when the request comes from an agent.

Why it matters

AI agents strip HTTP headers before the model sees content. The model never sees Content-Type, status codes, or redirect chains — it only sees the body text. Serving HTML forces agents to parse DOM structure, extract text, and discard layout — wasting tokens and losing meaning.

What we check

Agent UA gets non-HTML — We send a request with a known AI agent User-Agent (e.g. ClaudeBot) and check if the response Content-Type is something other than text/html. Sites that detect agent UAs and serve text/markdown, text/plain, or application/json pass this check.

Accept: JSON returns JSON — We send Accept: application/json and check if the response is valid JSON. This lets programmatic agents access structured data directly.

Accept: text returns text — We send Accept: text/plain and check if the response is plain text or markdown. This is the simplest format for language models to consume.

Accept: markdown returns markdown — We send Accept: text/markdown and check if the response is markdown content (Content-Type text/markdown or a markdown body). This is the format agents prefer when they want structure (headings, lists, links) without HTML weight.

How to implement

Detect agent User-Agents and Accept headers in your server middleware:

app.get('/', (req, res) => {
  const ua = (req.get('user-agent') || '').toLowerCase();
  const accept = req.get('accept') || '';
  const isAgent = /claude|gpt|anthropic|perplexity|gptbot/i.test(ua);

  // Markdown takes precedence when explicitly requested
  if (accept.includes('text/markdown')) {
    return res.type('text/markdown').sendFile('[llms.txt](/kb/llms-txt)');
  }

  if (accept.includes('application/json')) {
    return res.json({ name: 'My Service', api: '/openapi.json' });
  }

  if (isAgent || accept.includes('text/plain')) {
    return res.type('text/plain').sendFile('llms.txt');
  }

  res.sendFile('index.html');
});

Markdown specifics

Accept: text/markdown is a newer convention (popularized by the llms.txt ecosystem) and not yet universally implemented. Returning markdown when requested means:

The body is valid markdown (headings, lists, links — no HTML tags)
The Content-Type header is text/markdown (or text/plain with markdown body)
The same URL serving HTML to a browser can serve markdown to an agent — the path doesn't change, only the representation

For documentation pages, blog posts, and content pages, you typically already have a markdown source. Serve that source directly when Accept: text/markdown is sent, instead of converting to HTML first.

Spec maturity

Established. Content negotiation is defined in HTTP itself (RFC 9110). The convention of returning agent-friendly formats based on User-Agent or Accept is widely used by AI-aware sites.

Learn more

RFC 9110 §12: Content Negotiation
llmstxt.org — Markdown-for-LLMs convention

OpenAPI

agentgrade