## What is content negotiation?

Content negotiation means serving different responses based on what the client asks for. For AI agents, this means returning structured text (JSON, markdown, or plain text) instead of heavy HTML when the request comes from an agent.

## Why it matters

AI agents strip HTTP headers before the model sees content. The model never sees Content-Type, status codes, or redirect chains — it only sees the body text. Serving HTML forces agents to parse DOM structure, extract text, and discard layout — wasting tokens and losing meaning.

## What we check

**Agent UA gets non-HTML** — We send a request with a known AI agent User-Agent (e.g. `ClaudeBot`) and check if the response Content-Type is something other than `text/html`. Sites that detect agent UAs and serve `text/markdown`, `text/plain`, or `application/json` pass this check.

**Accept: JSON returns JSON** — We send `Accept: application/json` and check if the response is valid JSON. This lets programmatic agents access structured data directly.

**Accept: text returns text** — We send `Accept: text/plain` and check if the response is plain text or markdown. This is the simplest format for language models to consume.

**Accept: markdown returns markdown** — We send `Accept: text/markdown` and check if the response is markdown content (Content-Type `text/markdown` or a markdown body). This is the format agents prefer when they want structure (headings, lists, links) without HTML weight.

## How to implement

Detect agent User-Agents and `Accept` headers in your server middleware:

```javascript
app.get('/', (req, res) => {
  const ua = (req.get('user-agent') || '').toLowerCase();
  const accept = req.get('accept') || '';
  const isAgent = /claude|gpt|anthropic|perplexity|gptbot/i.test(ua);

  // Markdown takes precedence when explicitly requested
  if (accept.includes('text/markdown')) {
    return res.type('text/markdown').sendFile('[llms.txt](/kb/llms-txt)');
  }

  if (accept.includes('application/json')) {
    return res.json({ name: 'My Service', api: '/openapi.json' });
  }

  if (isAgent || accept.includes('text/plain')) {
    return res.type('text/plain').sendFile('llms.txt');
  }

  res.sendFile('index.html');
});
```

### Markdown specifics

`Accept: text/markdown` is a newer convention (popularized by the llms.txt ecosystem) and not yet universally implemented. Returning markdown when requested means:

- The body is valid markdown (headings, lists, links — no HTML tags)
- The Content-Type header is `text/markdown` (or `text/plain` with markdown body)
- The same URL serving HTML to a browser can serve markdown to an agent — the path doesn't change, only the representation

For documentation pages, blog posts, and content pages, you typically already have a markdown source. Serve that source directly when `Accept: text/markdown` is sent, instead of converting to HTML first.

## Spec maturity

**Established.** Content negotiation is defined in HTTP itself ([RFC 9110](https://www.rfc-editor.org/rfc/rfc9110)). The convention of returning agent-friendly formats based on User-Agent or Accept is widely used by AI-aware sites.

## Learn more

- [RFC 9110 §12: Content Negotiation](https://www.rfc-editor.org/rfc/rfc9110#name-content-negotiation)
- [llmstxt.org](https://llmstxt.org/) — Markdown-for-LLMs convention

## Related

- [OpenAPI](/kb/openapi)