---
title: "llms.txt: four ideas wearing the same filename"
description: "Target uses it as a store map. Etsy uses it to tell AI what not to say. Shopify uses it to teach agents how to check out. Twilio uses 2.5 MB to list everything. They all follow the same spec, and almost nobody reads any of it."
date: 2026-11-08T00:00:00.000Z
tags: ["llms", "agents", "markdown", "web"]
canonical: https://nick.rivett.dev/posts/llms-txt-four-ideas/
---

# llms.txt: four ideas wearing the same filename

`llms.txt` is four different ideas wearing the same filename.

The smallest file I found was 1.7 KB. [Svelte](https://svelte.dev/llms.txt) uses it to list seven documentation pages. The largest was 2.5 MB. [Twilio](https://www.twilio.com/llms.txt) uses it to list 8,201, and when I checked, its first H1 was on line 8,535. The spec only requires an H1, which is why the format stretches that far.

In between, the same file is a store map for [Target](https://www.target.com/llms.txt), a set of rules for [Etsy](https://www.etsy.com/llms.txt), a checkout tutorial for [Shopify stores](https://www.allbirds.com/llms.txt), and an index of indexes for [Cloudflare](https://developers.cloudflare.com/llms.txt). I read a pile of them from docs sites, blogs, and retailers. The format is one page. The uses have split into four. And the logs say almost nobody is reading any of it.

## The spec is almost aggressively small

Jeremy Howard published the first version in September 2024. [Version 2](https://llmstxt.org/changes.html) landed on 10 August 2026. It's a proposal on [llmstxt.org](https://llmstxt.org/), not a W3C or IETF standard.

The file goes at `/llms.txt` or any subpath. A file covers the URLs under its path. If several apply, the most specific one wins. `/docs/api/llms.txt` beats `/docs/llms.txt`, which beats `/llms.txt`.

The structure is an optional BOM, an H1, a blockquote summary, optional prose, and any number of H2 sections containing `[name](url)` links with optional notes after a colon. The only required part is the H1.

The links should point at LLM-friendly content, usually a `.md` twin. The file stays small. The detail lives behind the links.

v2 added two link relations for discovery. `rel="alternate" type="text/markdown"` points at a page's markdown version, and `rel="describedby"` points at the `llms.txt` that covers the page. Both can be HTML `<link>` elements or an HTTP `Link:` header. v2 also allowed `page.md` as well as `page.html.md`, and it made `Optional` a convention instead of a machine-readable skip list.

`llms-full.txt` is a companion convention, not part of the core syntax. It's the everything-in-one-file option.

That's the whole spec. That's why the same file can be a store map, a docs index, a checkout tutorial, and a list of things an AI must not say.

## One: the checkout tutorial

The Shopify stores I checked serve an `llms.txt` that is not about content at all.

[Allbirds' file](https://www.allbirds.com/llms.txt) is 5.8 KB. It opens with "You can browse products and shop at Allbirds's online store on your user's behalf." Then it explains the Shop skill, UCP discovery at `/.well-known/ucp`, the MCP endpoint at `/api/ucp/mcp`, WebMCP tools, buyer authorization, rate limits, and the store policies. It doesn't name a single shoe.

[Kylie Cosmetics](https://www.kyliecosmetics.com/llms.txt) serves the same file with the name swapped. It is platform boilerplate, not something the brand wrote. The canonical version is [`/agents.md`](https://www.allbirds.com/agents.md), and `/llms.txt` mirrors it.

This is the job that makes the `AGENTS.md` comparison feel right. The file is not a reading list. It is a manual for operating in a domain. It tells the agent which tools exist, how to call them, when to stop, and when to ask the human for approval. If you have written an `AGENTS.md` for a repo, the shape is familiar. Different domain, same idea.

## Two: the docs index

Most of the files I read are this: a map to clean markdown for docs, blogs, and support content.

The good ones are curated. Svelte's is 1.7 KB and seven links. [Vercel's](https://vercel.com/llms.txt) is 5.4 KB and 28 links, nearly all with a note. Cloudflare's is 16.8 KB and 112 links, but each link points at a per-product `llms.txt`, so the root stays small and the detail lives one hop away.

The bad ones are sitemaps with extra steps. Twilio's is 2.5 MB and 8,201 links. [Stripe's product file](https://stripe.com/llms.txt) is 69.9 KB and 308 links. Its [docs file](https://docs.stripe.com/llms.txt) is 99 KB. [Anthropic's](https://platform.claude.com/llms.txt) is 98 KB and 887 links. At that size, the file is not a summary. It is the site.

The files that work stay small. A short index, a blockquote that says what the thing is, and links that point at markdown. The ones that fail try to fit the whole site into one context window.

## Three: the store map

Retail is where the format gets interesting, because a store has a different problem. The agent needs categories, policies, and enough product context to answer a question, but prices and stock change too fast to live in a static file.

Target's file is 6.2 KB and 54 links. Core shopping categories, services and fulfillment, owned brands, and two `Optional` sections. It reads like the directory board at the entrance of a shop.

[Appliances Online](https://www.appliancesonline.com.au/llms.txt) is 17.2 KB and 89 links. Categories, popular product types, delivery and installation, price and returns, buying guides, product care, brands, deals. It also says the categories are "a curated selection of key pages and do not represent the full range available on the website." That is the right instinct.

[Nomad](https://www.nomadgoods.com/llms.txt) goes the other way. Its file is 105 KB and 457 links, with a long product list. [Fashion Nova's](https://www.fashionnova.com/llms.txt) is 5.5 KB and 58 links, closer to Target's shape.

The useful rule for a store is to point the index at categories, policies, and guides, and leave price and stock to the product page and the feed. An agent that reads a stale price from a static file is worse than an agent that fetches the product page.

## Four: the guardrail

Etsy's file is the one I keep coming back to. It's 26.7 KB, 76 links, and it isn't a catalog. It's a set of rules for answering questions about Etsy.

It starts with "What you can state confidently" and "What you must not assert". The second section is the interesting one. Do not invent listing IDs or shop names. Do not invent category slugs. Do not state prices, stock, review counts, shipping times, badge status, or discount claims from memory. Quote them only from a live page you are currently viewing. Do not quote an Etsy-wide return window, because returns are set per shop.

Then it routes. Buyer requests before the purchase, buyer requests after the purchase, seller requests. Every question belongs in exactly one place.

This is answer governance. It isn't a map to content, and it isn't a tool interface. It's a policy document that happens to live at `/llms.txt`. For a marketplace with millions of sellers and constantly changing prices, it might be the most useful version of the format I have seen.

## What it is not

`llms.txt` isn't a crawl control. It's a reading list. It has no allow or disallow grammar and no standards body behind it. robots.txt is the closest thing to a gate, but it's also a suggestion. Google and Bing respect it. Plenty of scrapers don't. If you want to signal your preferences to the crawlers that care, use robots.txt, `X-Robots-Tag`, or a content signal. Don't put crawl rules in a file that has no crawl rules.

The `AGENTS.md` comparison is better, but only for two of the four jobs. For the checkout tutorial and the guardrail, `llms.txt` really is the web's `AGENTS.md`. For the docs index and the store map, it is a map. The filename does not tell you which one you are reading.

## How is anyone supposed to find it?

The spec says agents are expected to view or search the file, but it only gives you two discovery mechanisms.

The first is the path. Try `/llms.txt` at the root, or the subpath you are already working in. The second is the link relations. A page points at its markdown twin with `rel="alternate" type="text/markdown"` and at the covering index with `rel="describedby"`. Both can be HTML `<link>` elements or an HTTP `Link:` header.

There is no `/.well-known/` location. The author rejected it because `/.well-known/` only exists at the site root, and `llms.txt` is path-scoped.

In practice, almost nobody implements the discovery half. [One August 2026 sample](https://llmtxt.info/blog/llms-txt-discovery-relations/) found 31 of 286 pages advertising a markdown alternate, and four advertising `describedby`. [Another check](https://llms-full-txt.ru/en/specification/discovery/) found `describedby` on one site out of 46, Cursor.

And no bot goes looking. Ahrefs saw zero AI requests to `llms.txt` files that did not exist. The file only gets read when something already knows to look. If you publish the file and skip the link relations, you have published a file for people who already know the URL.

## The readership problem

[Ahrefs](https://ahrefs.com/blog/llmstxt-study/) looked at 137,210 domains. 28% publish an `llms.txt`. 97% of those files got zero requests in May 2026.

[Arrivl](https://arrivl.ai/research/does-anything-fetch-your-llms-txt) read logs from 39 sites over 46 days. It found 100 requests to `/llms.txt` against 195,245 requests to ordinary pages. Markdown twins did better, but 7,993 of 8,179 twin requests were bingbot on one site. Arrivl put it well: the file is usually present and usually unrequested.

[Google's guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) to AI features says Search ignores the file. "Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." No major AI provider documents reading a third party's `llms.txt`. The clients that do fetch it are coding agents and tools like Context7 and LangChain's `mcpdoc`, usually because a human pointed them at it.

The caveats matter. The Ahrefs sample skews technical and SEO-aware. Arrivl's sites are small and mid-size businesses, not blogs. Fetching is not the same as using. But the direction is hard to miss.

## So why did I build one?

I built a markdown twin for this blog anyway. The build gets its own post. Turns out the endpoint is easy and the MDX components are not.

The short version is that I don't expect a bot to stumble onto this blog. I want to be able to hand an agent a URL and have it read the post instead of the nav, the footer, and the analytics script. That's a tool for a rainy day. It's also cheap, and I wanted to see if it would work.

The agent-readable web, at least right now, is four jobs, one filename, and a lot of unread files. I built one anyway. The robots can read the source now, if they ever show up.