A markdown twin for a rainy day
I don’t expect an agent to stumble onto this blog. I want to be able to hand one a URL on a rainy day and have it read the article, not the page around it. Take my 25 KB post about mermaid diagrams. Its markdown twin is 8 KB. For an LLM, that’s 17 KB less markup to parse.
There is no business case for this post. I built the twin anyway.
In part one I looked at the llms.txt format and the four jobs people have found for it. This post is about the build.
MDX is not markdown
This blog is built with Astro. It’s a static site generator, so the whole site is rendered at deploy time and served as files. The posts are MDX, which is markdown with JSX components mixed in. That’s how a post can contain an interactive Mermaid editor or a Blazor benchmark. It also means the source file isn’t plain markdown.
The posts use three custom components so far: a Mermaid editor in two, a Blazor benchmark in one, and an image in one. The source files are markdown with imports at the top and those components dropped into the prose. More will follow, and each one needs its own rule in the serializer.
If I copied the source to a .md file, an agent would get the prose plus a pile of imports and JSX. A generic markdown exporter would either skip the post entirely or strip the components and lose the diagrams.
So the twin is generated from the same source, with a serializer that knows what to do with each component.
The serializer
The endpoint is a prerendered Astro route at src/pages/posts/[...slug].md.ts. It reads the collection, calls postToMarkdown, and returns a Response with text/markdown; charset=utf-8. No runtime. The build writes a .md file next to each post’s HTML.
postToMarkdown does the work. The first problem is imports. A naive regex will delete any line starting with import, including a line inside a fenced code block. A future post about JavaScript would lose its examples. So the serializer splits the body into fenced and unfenced segments and only strips imports from the unfenced parts.
The second problem is components. The Mermaid editor is the interesting one. The rendered HTML has an empty textarea and the diagram source hidden in a data-example attribute. A generic HTML-to-markdown converter would drop it. The MDX source has the diagram in a prop, so the serializer pulls it out and writes a mermaid fence:
const MERMAID_EDITOR = /<MermaidEditor\s+code=\{`([\s\S]*?)`\}\s*\/>/g;
body.replace(MERMAID_EDITOR, (_, code) => "```mermaid\n" + code.trim() + "\n```");The MDX source:
<MermaidEditor
code={`graph LR
A[dev.mmd] --> B{git commit}
`}
/>What an agent gets at the same spot in the .md:
graph LR
A[dev.mmd] --> B{git commit}An agent can read or render the diagram source, and the page gets the interactive editor. Both come from the same code prop.
The benchmark is the opposite. It’s a Blazor WASM app running in an iframe. Markdown files can’t run WASM, so the serializer drops it and writes a blockquote that points back to the live post.
The image component becomes . The serializer resolves the frontmatter image through getImage and writes the absolute URL. If a post has an inline image that is not the frontmatter image, it falls back to alt text. One post has an image today, so the simple version is enough.
Any component without a rule gets dropped, and the build logs a warning with its name. That is the guard for the next component I add and forget about.
The headers
The layout emits <link rel="alternate" type="text/markdown" href="/posts/<id>.md"> on every post, plus <link rel="describedby" href="/llms.txt"> site-wide. public/_headers sets the content type and the robots header:
/*.md
Content-Type: text/markdown; charset=utf-8
X-Robots-Tag: noindexnoindex is deliberate. Disallow would stop Google from fetching the file, which means it never sees the noindex. If a .md URL gets linked somewhere, Google can still list the bare URL because it cannot fetch it. Leaving the file crawlable means Google fetches it once, sees the noindex, and drops it. AI agents ignore noindex for fetching, so the twins stay readable.
That’s the bit people get wrong. noindex isn’t a crawl block. It’s an indexing instruction.
The llms.txt index is a prerendered route too. It reads the collection, drops drafts, sorts newest first, and links each post’s .md. It’s about 20 lines of output. I skipped llms-full.txt. It’s optional, and one file per post is enough for this use case.
The tests
The tests are more code than the feature. I’m okay with that.
scripts/check-markdown-twins.mjs runs after the build in Woodpecker. It checks that every post has a .md, that known posts kept their content, that no imports or JSX leaked, that llms.txt links published posts and excludes drafts, and that the headers and sitemap are correct. tests/e2e/markdown-twins.spec.ts checks the head link on a few posts and fetches the .md files over HTTP.
The build-time check is the one that matters. The failure mode is silent: a new component leaks into the output and the file still returns 200. The check catches it before the deploy.
Why do it
I have no evidence that an agent has ever fetched one of these files. GoatCounter doesn’t track .md requests, so I can’t tell. The numbers from part one suggest the answer is no.
I built it anyway. The whole thing is a route, a serializer, two headers, and a test. It costs nothing to keep. It also forced me to make the content structure explicit. The Mermaid source is now machine-readable, which is a nice side effect I did not plan for.
And on the rainy day when I want an agent to know how I did something, I can hand it the URL and get the article instead of the page around it. That’s the whole bet.
If an agent shows up, it gets the good version. If not, I still have a clean markdown copy of everything I wrote. That’s a fine trade for an afternoon.