robots.txt, sitemap.xml, llms.txt: Three Files Machines Read
Every website holds three conversations with machines, and most sites prepare for only one. robots.txt says what may be fetched. sitemap.xml says what exists. llms.txt — the newest of the three, and by far the most oversold — says what matters, in a form a model can read without fighting your navigation.
Two of those files are settled engineering. The third is a proposal with a marketing problem: a convention most AI crawlers have never requested. Keeping them apart is the quickest way to stop paying for SEO theatre.
In short
- robots.txt is a real standard now — RFC 9309, September 2022 — and it governs crawling, never indexing.
- A blocked page can still be indexed. Google documents it plainly;
noindexis what removes pages, and it needs a crawlable page to work. - sitemap.xml buys discovery, not rankings. 50,000 URLs or 50 MB uncompressed per file; more means a sitemap index.
lastmodis a promise. Google schedules crawls from it only while it consistently matches reality.- llms.txt is a 2024 proposal almost nobody fetches. Ahrefs measured 97% of published files receiving zero requests.
- The honest setup is cheap: a short robots.txt, a build‑generated sitemap with truthful dates, a curated llms.txt treated as documentation.
robots.txt: an RFC with a narrower job than people think
For twenty‑eight years robots.txt was a gentleman’s agreement between webmasters and spiders. In September 2022 it became RFC 9309, a Standards Track document authored by Martijn Koster with Google’s Gary Illyes, Henner Zeller and Lizzi Sassman. It fixes syntax, error handling and caching — and contains one sentence worth framing: these rules “are not a form of access authorization.”
That sentence kills the most popular robots.txt myth. The file controls crawling. Google’s own specification says content on disallowed pages can’t be indexed, “but it may still index the URL and show it in search results without a snippet.” If the URL is linked from anywhere else, it stays discoverable whatever your file claims. Removing a page takes noindex, and noindex only works if the crawler is allowed to fetch the page and see it — block the crawl and you block the instruction with it. The most common self‑inflicted wound we see in audits: hundreds of pages “deleted” in robots.txt, still indexed a year later.
The details that matter:
- Google parses exactly four fields —
user-agent,allow,disallow,sitemap. Everything else,crawl-delayincluded, is ignored. - Google enforces a 500 KiB cap, discards everything past it, and caches the file for up to 24 hours — so a mistake outlives the deploy that fixes it.
crawl-delayis heard by Bing and by Anthropic’s ClaudeBot, not by Google, where crawl rate lives in Search Console.
sitemap.xml: the discovery layer
The sitemaps.org protocol dates to 2005, published under a Creative Commons licence with support from Google, Yahoo and Microsoft. Its own homepage sets expectations correctly: the protocol “does not guarantee that web pages are included in search engines, but provides hints for web crawlers to do a better job.”
Hints, not votes: a sitemap helps a crawler find pages it would otherwise miss — deep archive pages, thinly linked pages, fresh URLs nobody has linked yet. It never raised rankings.
The limits are hard:
- 50,000 URLs or 50 MB (52,428,800 bytes) uncompressed per file. Exceed either and the file is invalid.
- Beyond that, a sitemap index: up to 50,000 sitemaps, itself capped at 50 MB.
lastmod deserves its own paragraph, because it is the field most often faked and the one that pays when real. In June 2023 Google deprecated the sitemap ping endpoint and explained how lastmod is used: as a signal for scheduling crawls of already‑known URLs. The condition is honesty. “If your page changed 7 years ago, but you’re telling us in the lastmod element that it changed yesterday, eventually we’re not going to believe you anymore.” The same post settles two debates: Google uses neither changefreq nor priority.
So: generate the sitemap at build time, include only canonical URLs that return 200, stamp lastmod from the content’s real modification date rather than the build clock, and drop it where your CMS can’t tell the truth. Then announce it — a Sitemap: line in robots.txt, plus Search Console.
llms.txt: the new file and the honest expectations
In September 2024 Jeremy Howard of Answer.AI proposed a /llms.txt file: a markdown document giving a model a curated map of a site, with links to the pages worth reading, instead of making it reverse‑engineer a template full of navigation. The proposal now lives at llmstxt.org in a revised v2, updated in August 2026.
The format is deliberately small. An H1 with the site name — the only required element. A blockquote summary. Then optional lists under H2 headings shaped like - [Page title](url): one line about what's there, with a ## Optional section for links an agent may skip when context is tight. The original example, FastHTML, ships a full‑text companion most people call llms-full.txt. It can sit at the root or any subpath, covering everything beneath.
What it is not: a standard. No RFC, no vendor mandate, no parsing requirement. And the adoption data is brutal. In May 2026 Ahrefs analysed server logs from roughly 137,000 domains and found that 28% publish an llms.txt and 97% of those files received zero requests in a month. Zero requests also came from AI bots for files that don’t exist — they never go looking.
Google’s position is equally blunt. John Mueller wrote on Bluesky that “no AI system currently uses llms.txt”. In October 2026 he added that the file can’t stand in for an XML sitemap, that with llms.txt “the hope is bigger than the reality”, and that a site wanting AI crawlers to find it should keep the conventional sitemap.xml name or publish an RSS feed.
Why does it keep appearing in briefs? Partly tooling: llmstxt.org notes that Chrome’s Lighthouse audits for an llms.txt in its agentic browsing checks, so it shows up in reports whether or not anyone reads it. Partly because it is pleasant to write — the document you’d hand a new colleague on their first morning.
What actually gets you read by AI crawlers
Not llms.txt. Permissions and HTML. OpenAI manages GPTBot (training) and OAI‑SearchBot (search) through robots.txt groups; Anthropic says its bots honour “industry standard directives in robots.txt”; Perplexity documents the same for PerplexityBot. The nuance: user‑triggered fetchers are a different species. OpenAI notes ChatGPT‑User acts on a person’s request, so robots.txt rules “may not apply”; Perplexity says Perplexity‑User generally ignores them. “robots.txt is how you talk to AI crawlers” is true for the automated ones and false for the ones a human set loose on your page.
What an honest setup looks like
robots.txt — twenty lines: explicit groups for search crawlers and for the AI bots you’ve decided about, a Sitemap: line, nothing above 500 KiB, nothing you want indexed behind a Disallow.
sitemap.xml — generated from the build, canonicals only, lastmod from content changes, an index once you pass 50,000 URLs, submitted in Search Console and Bing Webmaster Tools.
llms.txt — curated, not generated. Thirty links with honest one‑line notes beat four thousand auto‑dumped URLs. Ship it, check your logs a quarter later; if nothing fetched it, you’ve confirmed the industry baseline and lost an hour.
Logs — the only measurement that doesn’t lie: which bots arrived, what they fetched, whether robots.txt was honoured. That’s how you learn whether any of this works.
Summary
The three files are not three versions of one idea. robots.txt is a standard that rations access; sitemap.xml is a protocol that lists URLs; llms.txt is a proposal describing a site for machines that mostly aren’t listening yet. Getting the first two right is table stakes and takes an afternoon. The third costs an hour and buys goodwill with the humans who read it — not a ranking, not a citation, and no shortcut past the fundamentals.
If your robots.txt is a museum of 2016 plugins, start with technical SEO — the crawlability, rendering and markup layer underneath all of this. If you’d rather have the whole stack audited and rebuilt, that’s our SEO & GEO service. And if you’re wondering whether any of it matters when answers replace clicks, we covered that separately: Is SEO dead in the AI era?. For how search engines judge visitor behaviour once humans arrive, see our piece on NavBoost and user interaction signals.
Yuri Osipov
Founder, SEO & analytics
Runs Take the Space. Specialises in technical SEO, measurable analytics setups and AI search visibility.