What is agent content negotiation?
Content negotiation means serving different responses based on what the client asks for. For AI agents, this means returning structured text (JSON, markdown, or plain text) instead of heavy HTML when the request comes from an agent.
Why it matters
AI agents strip HTTP headers before the model sees content. The model never sees Content-Type, status codes, or redirect chains — it only sees the body text. Serving HTML forces agents to parse DOM structure, extract text, and discard layout — wasting tokens and losing meaning.
Sites that detect agent User-Agents and serve clean text or JSON give agents exactly what they need.
What AgentGrade checks
Agent UA gets non-HTML — We send a request with a known AI agent User-Agent (e.g. ClaudeBot) and check if the response Content-Type is something other than text/html. Sites that detect agent UAs and serve text/markdown, text/plain, or application/json pass this check.
Accept: JSON returns JSON — We send Accept: application/json and check if the response is valid JSON. This lets programmatic agents access structured data directly.
Accept: text returns text — We send Accept: text/plain and check if the response is plain text or markdown. This is the simplest format for language models to consume.
How to implement
Detect agent User-Agents and Accept headers in your server middleware:
const AGENT_UA = /ClaudeBot|ChatGPT-User|GPTBot|PerplexityBot|Gemini/i;
app.use((req, res, next) => {
const ua = req.headers['user-agent'] || '';
const accept = req.headers['accept'] || '';
if (AGENT_UA.test(ua) && !accept.includes('text/html')) {
return res.type('text/plain').sendFile('llms.txt');
}
if (accept.includes('application/json') && !accept.includes('text/html')) {
return res.json({ name: 'My Service', description: '...' });
}
if (accept === 'text/plain' || accept === 'text/markdown') {
return res.type('text/plain').sendFile('llms.txt');
}
next();
});
Known AI agent User-Agents
| Agent | User-Agent string |
|-------|-------------------|
| Claude | ClaudeBot/1.0 |
| ChatGPT | ChatGPT-User |
| GPTBot | GPTBot/1.0 |
| Perplexity | PerplexityBot |
| Gemini | Google-Extended |