Yuri Osipov · Founder, SEO & analytics

robots.txt, sitemap.xml, llms.txt: Three Files Machines Read

Every website holds three conversations with machines, and most sites prepare for only one. robots.txt says what may be fetched. sitemap.xml says what exists. llms.txt — the newest of the three, and by far the most oversold — says what matters, in a form a model can read without fighting your navigation.

Two of those files are settled engineering. The third is a proposal with a marketing problem: a convention most AI crawlers have never requested. Keeping them apart is the quickest way to stop paying for SEO theatre.

In short

robots.txt: an RFC with a narrower job than people think

For twenty‑eight years robots.txt was a gentleman’s agreement between webmasters and spiders. In September 2022 it became RFC 9309, a Standards Track document authored by Martijn Koster with Google’s Gary Illyes, Henner Zeller and Lizzi Sassman. It fixes syntax, error handling and caching — and contains one sentence worth framing: these rules “are not a form of access authorization.”

That sentence kills the most popular robots.txt myth. The file controls crawling. Google’s own specification says content on disallowed pages can’t be indexed, “but it may still index the URL and show it in search results without a snippet.” If the URL is linked from anywhere else, it stays discoverable whatever your file claims. Removing a page takes noindex, and noindex only works if the crawler is allowed to fetch the page and see it — block the crawl and you block the instruction with it. The most common self‑inflicted wound we see in audits: hundreds of pages “deleted” in robots.txt, still indexed a year later.

The details that matter:

sitemap.xml: the discovery layer

The sitemaps.org protocol dates to 2005, published under a Creative Commons licence with support from Google, Yahoo and Microsoft. Its own homepage sets expectations correctly: the protocol “does not guarantee that web pages are included in search engines, but provides hints for web crawlers to do a better job.”

Hints, not votes: a sitemap helps a crawler find pages it would otherwise miss — deep archive pages, thinly linked pages, fresh URLs nobody has linked yet. It never raised rankings.

The limits are hard:

lastmod deserves its own paragraph, because it is the field most often faked and the one that pays when real. In June 2023 Google deprecated the sitemap ping endpoint and explained how lastmod is used: as a signal for scheduling crawls of already‑known URLs. The condition is honesty. “If your page changed 7 years ago, but you’re telling us in the lastmod element that it changed yesterday, eventually we’re not going to believe you anymore.” The same post settles two debates: Google uses neither changefreq nor priority.

So: generate the sitemap at build time, include only canonical URLs that return 200, stamp lastmod from the content’s real modification date rather than the build clock, and drop it where your CMS can’t tell the truth. Then announce it — a Sitemap: line in robots.txt, plus Search Console.

llms.txt: the new file and the honest expectations

In September 2024 Jeremy Howard of Answer.AI proposed a /llms.txt file: a markdown document giving a model a curated map of a site, with links to the pages worth reading, instead of making it reverse‑engineer a template full of navigation. The proposal now lives at llmstxt.org in a revised v2, updated in August 2026.

The format is deliberately small. An H1 with the site name — the only required element. A blockquote summary. Then optional lists under H2 headings shaped like - [Page title](url): one line about what's there, with a ## Optional section for links an agent may skip when context is tight. The original example, FastHTML, ships a full‑text companion most people call llms-full.txt. It can sit at the root or any subpath, covering everything beneath.

What it is not: a standard. No RFC, no vendor mandate, no parsing requirement. And the adoption data is brutal. In May 2026 Ahrefs analysed server logs from roughly 137,000 domains and found that 28% publish an llms.txt and 97% of those files received zero requests in a month. Zero requests also came from AI bots for files that don’t exist — they never go looking.

Google’s position is equally blunt. John Mueller wrote on Bluesky that “no AI system currently uses llms.txt”. In October 2026 he added that the file can’t stand in for an XML sitemap, that with llms.txt “the hope is bigger than the reality”, and that a site wanting AI crawlers to find it should keep the conventional sitemap.xml name or publish an RSS feed.

Why does it keep appearing in briefs? Partly tooling: llmstxt.org notes that Chrome’s Lighthouse audits for an llms.txt in its agentic browsing checks, so it shows up in reports whether or not anyone reads it. Partly because it is pleasant to write — the document you’d hand a new colleague on their first morning.

What actually gets you read by AI crawlers

Not llms.txt. Permissions and HTML. OpenAI manages GPTBot (training) and OAI‑SearchBot (search) through robots.txt groups; Anthropic says its bots honour “industry standard directives in robots.txt”; Perplexity documents the same for PerplexityBot. The nuance: user‑triggered fetchers are a different species. OpenAI notes ChatGPT‑User acts on a person’s request, so robots.txt rules “may not apply”; Perplexity says Perplexity‑User generally ignores them. “robots.txt is how you talk to AI crawlers” is true for the automated ones and false for the ones a human set loose on your page.

What an honest setup looks like

robots.txt — twenty lines: explicit groups for search crawlers and for the AI bots you’ve decided about, a Sitemap: line, nothing above 500 KiB, nothing you want indexed behind a Disallow.

sitemap.xml — generated from the build, canonicals only, lastmod from content changes, an index once you pass 50,000 URLs, submitted in Search Console and Bing Webmaster Tools.

llms.txt — curated, not generated. Thirty links with honest one‑line notes beat four thousand auto‑dumped URLs. Ship it, check your logs a quarter later; if nothing fetched it, you’ve confirmed the industry baseline and lost an hour.

Logs — the only measurement that doesn’t lie: which bots arrived, what they fetched, whether robots.txt was honoured. That’s how you learn whether any of this works.

Summary

The three files are not three versions of one idea. robots.txt is a standard that rations access; sitemap.xml is a protocol that lists URLs; llms.txt is a proposal describing a site for machines that mostly aren’t listening yet. Getting the first two right is table stakes and takes an afternoon. The third costs an hour and buys goodwill with the humans who read it — not a ranking, not a citation, and no shortcut past the fundamentals.

If your robots.txt is a museum of 2016 plugins, start with technical SEO — the crawlability, rendering and markup layer underneath all of this. If you’d rather have the whole stack audited and rebuilt, that’s our SEO & GEO service. And if you’re wondering whether any of it matters when answers replace clicks, we covered that separately: Is SEO dead in the AI era?. For how search engines judge visitor behaviour once humans arrive, see our piece on NavBoost and user interaction signals.

Yuri Osipov

Yuri Osipov

Founder, SEO & analytics

Runs Take the Space. Specialises in technical SEO, measurable analytics setups and AI search visibility.

All posts by the author →
Does blocking a page in robots.txt remove it from Google?
No. robots.txt controls crawling, not indexing. Google’s documentation states that a disallowed page can’t have its content indexed, but the URL may still appear in results without a snippet. To actually remove a page you need a noindex directive — and noindex only works if the crawler is allowed to fetch the page and read it.
How large can robots.txt and sitemap.xml be?
Google enforces a 500 KiB limit on robots.txt and silently ignores everything past it. A sitemap file may contain at most 50,000 URLs and be no larger than 50 MB (52,428,800 bytes) uncompressed. Beyond that you split the site across several sitemaps and list them in a sitemap index file, which itself is capped at 50,000 sitemaps.
Should we publish an llms.txt file?
Yes, if you treat it as documentation rather than a growth lever. It costs an hour and helps the humans and agents who actually land on it. But do not sell it as an AI visibility trick: Ahrefs measured 97% of llms.txt files receiving zero requests in May 2026, and Google says it doesn’t use the file at all.
Does the crawl‑delay directive work?
Not on Google. Its parser reads only user‑agent, allow, disallow and sitemap, so crawl‑delay is ignored — crawl rate there is set in Search Console. Bing supports crawl‑delay, and Anthropic’s bots support it as well, so the directive is not useless, just not universal.
Do AI crawlers read robots.txt?
The automated ones do: OpenAI’s GPTBot and OAI‑SearchBot, Anthropic’s ClaudeBot and Perplexity’s PerplexityBot are all managed through robots.txt groups. User‑triggered fetchers are the exception — ChatGPT‑User and Perplexity‑User fetch a page because a person asked for it, so robots.txt rules generally don’t apply to them.

Want a high‑performance website?

Let’s discuss how we can help your business grow.