llms.txt: four ideas wearing the same filename
llms.txt is four different ideas wearing the same filename.
The smallest file I found was 1.7 KB. Svelte uses it to list seven documentation pages. The largest was 2.5 MB. Twilio uses it to list 8,201, and when I checked, its first H1 was on line 8,535. The spec only requires an H1, which is why the format stretches that far.
In between, the same file is a store map for Target, a set of rules for Etsy, a checkout tutorial for Shopify stores, and an index of indexes for Cloudflare. I read a pile of them from docs sites, blogs, and retailers. The format is one page. The uses have split into four. And the logs say almost nobody is reading any of it.
The spec is almost aggressively small
Jeremy Howard published the first version in September 2024. Version 2 landed on 10 August 2026. It’s a proposal on llmstxt.org, not a W3C or IETF standard.
The file goes at /llms.txt or any subpath. A file covers the URLs under its path. If several apply, the most specific one wins. /docs/api/llms.txt beats /docs/llms.txt, which beats /llms.txt.
The structure is an optional BOM, an H1, a blockquote summary, optional prose, and any number of H2 sections containing [name](url) links with optional notes after a colon. The only required part is the H1.
The links should point at LLM-friendly content, usually a .md twin. The file stays small. The detail lives behind the links.
v2 added two link relations for discovery. rel="alternate" type="text/markdown" points at a page’s markdown version, and rel="describedby" points at the llms.txt that covers the page. Both can be HTML <link> elements or an HTTP Link: header. v2 also allowed page.md as well as page.html.md, and it made Optional a convention instead of a machine-readable skip list.
llms-full.txt is a companion convention, not part of the core syntax. It’s the everything-in-one-file option.
That’s the whole spec. That’s why the same file can be a store map, a docs index, a checkout tutorial, and a list of things an AI must not say.
One: the checkout tutorial
The Shopify stores I checked serve an llms.txt that is not about content at all.
Allbirds’ file is 5.8 KB. It opens with “You can browse products and shop at Allbirds’s online store on your user’s behalf.” Then it explains the Shop skill, UCP discovery at /.well-known/ucp, the MCP endpoint at /api/ucp/mcp, WebMCP tools, buyer authorization, rate limits, and the store policies. It doesn’t name a single shoe.
Kylie Cosmetics serves the same file with the name swapped. It is platform boilerplate, not something the brand wrote. The canonical version is /agents.md, and /llms.txt mirrors it.
This is the job that makes the AGENTS.md comparison feel right. The file is not a reading list. It is a manual for operating in a domain. It tells the agent which tools exist, how to call them, when to stop, and when to ask the human for approval. If you have written an AGENTS.md for a repo, the shape is familiar. Different domain, same idea.
Two: the docs index
Most of the files I read are this: a map to clean markdown for docs, blogs, and support content.
The good ones are curated. Svelte’s is 1.7 KB and seven links. Vercel’s is 5.4 KB and 28 links, nearly all with a note. Cloudflare’s is 16.8 KB and 112 links, but each link points at a per-product llms.txt, so the root stays small and the detail lives one hop away.
The bad ones are sitemaps with extra steps. Twilio’s is 2.5 MB and 8,201 links. Stripe’s product file is 69.9 KB and 308 links. Its docs file is 99 KB. Anthropic’s is 98 KB and 887 links. At that size, the file is not a summary. It is the site.
The files that work stay small. A short index, a blockquote that says what the thing is, and links that point at markdown. The ones that fail try to fit the whole site into one context window.
Three: the store map
Retail is where the format gets interesting, because a store has a different problem. The agent needs categories, policies, and enough product context to answer a question, but prices and stock change too fast to live in a static file.
Target’s file is 6.2 KB and 54 links. Core shopping categories, services and fulfillment, owned brands, and two Optional sections. It reads like the directory board at the entrance of a shop.
Appliances Online is 17.2 KB and 89 links. Categories, popular product types, delivery and installation, price and returns, buying guides, product care, brands, deals. It also says the categories are “a curated selection of key pages and do not represent the full range available on the website.” That is the right instinct.
Nomad goes the other way. Its file is 105 KB and 457 links, with a long product list. Fashion Nova’s is 5.5 KB and 58 links, closer to Target’s shape.
The useful rule for a store is to point the index at categories, policies, and guides, and leave price and stock to the product page and the feed. An agent that reads a stale price from a static file is worse than an agent that fetches the product page.
Four: the guardrail
Etsy’s file is the one I keep coming back to. It’s 26.7 KB, 76 links, and it isn’t a catalog. It’s a set of rules for answering questions about Etsy.
It starts with “What you can state confidently” and “What you must not assert”. The second section is the interesting one. Do not invent listing IDs or shop names. Do not invent category slugs. Do not state prices, stock, review counts, shipping times, badge status, or discount claims from memory. Quote them only from a live page you are currently viewing. Do not quote an Etsy-wide return window, because returns are set per shop.
Then it routes. Buyer requests before the purchase, buyer requests after the purchase, seller requests. Every question belongs in exactly one place.
This is answer governance. It isn’t a map to content, and it isn’t a tool interface. It’s a policy document that happens to live at /llms.txt. For a marketplace with millions of sellers and constantly changing prices, it might be the most useful version of the format I have seen.
What it is not
llms.txt isn’t a crawl control. It’s a reading list. It has no allow or disallow grammar and no standards body behind it. robots.txt is the closest thing to a gate, but it’s also a suggestion. Google and Bing respect it. Plenty of scrapers don’t. If you want to signal your preferences to the crawlers that care, use robots.txt, X-Robots-Tag, or a content signal. Don’t put crawl rules in a file that has no crawl rules.
The AGENTS.md comparison is better, but only for two of the four jobs. For the checkout tutorial and the guardrail, llms.txt really is the web’s AGENTS.md. For the docs index and the store map, it is a map. The filename does not tell you which one you are reading.
How is anyone supposed to find it?
The spec says agents are expected to view or search the file, but it only gives you two discovery mechanisms.
The first is the path. Try /llms.txt at the root, or the subpath you are already working in. The second is the link relations. A page points at its markdown twin with rel="alternate" type="text/markdown" and at the covering index with rel="describedby". Both can be HTML <link> elements or an HTTP Link: header.
There is no /.well-known/ location. The author rejected it because /.well-known/ only exists at the site root, and llms.txt is path-scoped.
In practice, almost nobody implements the discovery half. One August 2026 sample found 31 of 286 pages advertising a markdown alternate, and four advertising describedby. Another check found describedby on one site out of 46, Cursor.
And no bot goes looking. Ahrefs saw zero AI requests to llms.txt files that did not exist. The file only gets read when something already knows to look. If you publish the file and skip the link relations, you have published a file for people who already know the URL.
The readership problem
Ahrefs looked at 137,210 domains. 28% publish an llms.txt. 97% of those files got zero requests in May 2026.
Arrivl read logs from 39 sites over 46 days. It found 100 requests to /llms.txt against 195,245 requests to ordinary pages. Markdown twins did better, but 7,993 of 8,179 twin requests were bingbot on one site. Arrivl put it well: the file is usually present and usually unrequested.
Google’s guide to AI features says Search ignores the file. “Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” No major AI provider documents reading a third party’s llms.txt. The clients that do fetch it are coding agents and tools like Context7 and LangChain’s mcpdoc, usually because a human pointed them at it.
The caveats matter. The Ahrefs sample skews technical and SEO-aware. Arrivl’s sites are small and mid-size businesses, not blogs. Fetching is not the same as using. But the direction is hard to miss.
So why did I build one?
I built a markdown twin for this blog anyway. The build gets its own post. Turns out the endpoint is easy and the MDX components are not.
The short version is that I don’t expect a bot to stumble onto this blog. I want to be able to hand an agent a URL and have it read the post instead of the nav, the footer, and the analytics script. That’s a tool for a rainy day. It’s also cheap, and I wanted to see if it would work.
The agent-readable web, at least right now, is four jobs, one filename, and a lot of unread files. I built one anyway. The robots can read the source now, if they ever show up.