Infrastructure — Cache, Errors, Redirects

What is "Infrastructure"?

The Infrastructure group covers low-level HTTP behavior that determines whether agents can interact with your origin reliably. None of these are protocol-specific — they apply to any site, whether it serves an MCP endpoint, an OpenAPI spec, or just static content.

HTTPS redirect

If a human types yourdomain.com into a browser, the browser tries HTTPS first and most of the time gets there silently. Agents don't do that. They follow whatever URL they were handed — and if that URL is http://yourdomain.com and your server doesn't redirect, the agent's HTTP library either refuses to send the request (most modern AI clients block plain HTTP) or sends it unencrypted (older ones). Either way, the agent never reaches your real site.

Fix: respond with 301 or 308 from http://yourdomain.com/* to the same path on HTTPS. Most platforms do this for you (Cloudflare, Vercel, Heroku, build.io). If you self-host, add an nginx server block listening on port 80 that issues the redirect.

Sitemap.xml

Agents discover your URL surface from /sitemap.xml. Without it, the only paths an agent can reach are those linked from your homepage or declared in llms.txt — typically a small fraction of your real content.

Fix: serve an XML sitemap at /sitemap.xml. Frameworks like Next.js, Hugo, Jekyll, and Wordpress generate one automatically. For a hand-rolled sitemap:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://yourdomain.com/</loc></url>
  <url><loc>https://yourdomain.com/docs</loc></url>
</urlset>

Cache headers

Every well-configured server emits at least one of Cache-Control, ETag, or Last-Modified on responses. Without them, agents have no way to know whether to re-fetch or use a cached copy — they either re-download every time (slow, expensive) or stale-cache forever (incorrect data).

Fix: most web frameworks set these by default for static files. For dynamic responses, set Cache-Control explicitly:

// Express — for content that changes occasionally
res.set('Cache-Control', 'public, max-age=3600');

// For never-cache responses (still satisfies the check):
res.set('Cache-Control', 'no-store');

no-store counts as a valid cache header — it tells the agent "do not cache this," which is itself useful information. The check fails when *no* cache header is present at all.

Structured errors (JSON 404)

When an agent hits a wrong URL, an HTML 404 page (with nav, footer, search box, etc.) wastes tokens and can fool a confused agent into thinking it got real content. A JSON 404 returns a small, parseable error:

{ "error": "not_found", "path": "/wrong-url" }

Fix: use content negotiation in your 404 handler. Return JSON when the client sends Accept: application/json, HTML otherwise. Express example:

app.use((req, res) => {
  res.status(404);
  res.format({
    'application/json': () => res.json({ error: 'not_found', path: req.path }),
    'text/html': () => res.send('<h1>404 Not Found</h1>'),
    default: () => res.json({ error: 'not_found', path: req.path }),
  });
});

This change is invisible to humans (browsers send Accept: text/html and get HTML) but transformative for agents (they send Accept: application/json and get JSON they can parse).

Rate limit headers (optional)

If your service rate-limits, expose the state via X-RateLimit-Remaining, X-RateLimit-Reset, or Retry-After. Agents that hit a limit can wait the right amount of time and retry instead of failing.

This check is optional because most sites don't rate-limit at all — only API providers and SaaS apps do. Static sites have nothing to expose.

Why these are required

None of these break a website for humans. A site missing cache headers still loads in a browser; a site that returns HTML 404s still shows a friendly error page; a site with no sitemap still ranks in Google. That's exactly why they're easy to overlook — humans don't feel the problem.

Agents do. Every one of these is the difference between "the agent gets the data" and "the agent gets noise, wastes tokens, or gives up." Marking them required is how we keep them from being silently skipped.

agentgrade