September 27, 2026 · by Geoff
What an agent actually wants from a website
A short checklist for making a site useful to crawlers, assistants and coding agents without hurting it for people.
machine version: /machine/blog/what-an-agent-wants-from-a-website.md
I spent a few weekends looking at how the sites I own get consumed. The human traffic still comes from search and links. The rest, which is growing fast, is bots: model-training crawlers, retrieval fetchers behind assistants, and agents running tasks for a person. Here is what they want, roughly in order.
- A plain-text summary at a known URL.
/llms.txtis the convention that has stuck. Say who you are, what the site is for, list the key pages with one line each, and tell the reader what you would like it to do with the content. - Markdown, not HTML. A page that exists as markdown costs a fraction of the tokens and none of the boilerplate. Serve it with
text/markdownand link it from the HTML page withrel="alternate". - Structured data that is not lying. A schema.org
Person,OrganizationorBlogPostingblock that matches what the page says. Models use it to disambiguate you from the other people with your name. - JSON where there is a list. Projects, posts, products. If it is a list on the page, it should be an array at an endpoint.
- A discovery document. An agent card at
/.well-known/agent-card.jsonor an OpenAPI file at a predictable path means the agent does not have to guess. - A way to reach the human. Not a contact form; an endpoint that accepts a question and returns a ticket.
- A way to pay. HTTP 402 with an
x402payment requirement turns "please don't scrape me" into "here is the price".
None of this takes anything away from the human version of the site. It is the same content with the presentation removed, which is a healthy exercise on its own.