What Is an AI Agent?
The definition
An AI agent is software that uses a large language model to plan and execute actions on a user's behalf — running searches, calling APIs, comparing prices, filling orders, making payments. The defining trait is not intelligence but delegation: a human states a goal, and the agent decides which sites to visit, which tools to call, and when the job is done.
That makes an agent a fundamentally different visitor from anything the web was built for. It does not browse — it uses. It has no eyes for your design, no patience for your clickflow, and a hard timeout where a human would wait.
Agent vs. bot vs. crawler
The three words get used interchangeably; they describe different machines:
| Crawler | Bot | Agent | |
|---|---|---|---|
| Drives itself by | A schedule and a URL frontier | A fixed script | A goal and a plan |
| Works for | An index or dataset | Whoever wrote the script | One user, right now |
| Visits your site | Broadly and repeatedly | Wherever the script says | Only if you're useful to the task |
| Example | GPTBot, ClaudeBot | A price-monitoring script | ChatGPT booking a table |
The confusion is understandable because the same vendors run all three. OpenAI operates a training crawler (GPTBot), a search-index crawler (OAI-SearchBot), and the live fetcher its agents use (ChatGPT-User) — three tiers with separate robots.txt tokens and separate behavior. The robots.txt guide explains the tier model; the user-agent reference documents every bot in it.
The anatomy of an agent
Four parts, each of which meets your website differently:
- The planner — the LLM itself. It never touches your server; it only sees what the other parts bring back.
- The tools — how the agent acts. If your site exposes an MCP server or an OpenAPI spec, the agent can call your capabilities directly instead of scraping pages.
- The fetcher — the HTTP client that actually hits your server. This is where content negotiation happens: a fetcher that asks for
text/markdownand gets clean markdown ingests your content in one pass. - The wallet — how the agent pays. A machine-readable 402 payment challenge is the difference between an agent buying from you and an agent recommending your competitor.
What an agent needs from your website
An agent completes a task on your site exactly as far as your machine-readable surface lets it. The layers, in the order an agent encounters them: discovery files (llms.txt, agent-aware robots.txt) so it can orient; callable capabilities (MCP, OpenAPI) so it can act; content negotiation so it can read; and a payment path so it can transact. That stack is what agent readiness measures.
What agent traffic looks like in your logs
Less obvious than you'd expect. Some of it announces itself — the named user-agents documented in the reference, verifiable against published IP ranges. But a large share arrives anonymously: coding agents and desktop assistants fetch from the user's own machine, so the request comes from a residential IP with a generic client string like curl or a plain browser UA. If your logs show growing traffic from generic clients that fetch one page and never load assets, that is often agent traffic you cannot attribute. Blocking it blocks the humans who sent it.
FAQ
Is ChatGPT an AI agent?
ChatGPT is an assistant that contains agents. When it only converses, it isn't acting as one; when it searches the web, fetches your page, or completes a checkout on a user's behalf, that is agent behavior — and it arrives at your server as ChatGPT-User or an agent-mode fetcher.
Is an AI agent the same as a chatbot?
No. A chatbot converses; an agent acts. The test is whether the software takes real-world actions (fetching, calling APIs, transacting) in pursuit of a goal, not whether it talks.
Do AI agents render JavaScript?
Mostly no. Most fetchers read the raw HTTP response and never execute scripts, so content that only exists after JavaScript runs is invisible to them. See JavaScript rendering.
How do I know whether agents visit my site?
Check your access logs against the named tokens in the user-agent reference, and watch for the anonymous pattern: generic client strings, residential IPs, single-page fetches with no asset loads. Then scan your site to see what those visitors can actually do once they arrive.