Your Site Sends the Same HTML to Chrome and to Agents. The Fix Has Been in HTTP Since 1997.
A site called acceptmarkdown.com hit the Hacker News front page yesterday with a narrow argument: when an AI agent fetches your article, it should be able to say Accept: text/markdown and get clean Markdown back from the same URL that serves HTML to a browser. Not a parallel .md file at a different address. The same URL, negotiating.
The mechanism is proactive content negotiation, specified in RFC 9110 §12.5.1, and it has been in HTTP since the nineties doing the same job for Accept-Language. What is new is that there is finally a client population that wants a different media type from the same resource, and it is growing fast enough to be worth ten lines of config.
I run 506 posts as static HTML off a Hetzner box behind nginx. So this one is not a thinkpiece for me, it is a config change. Here is what I found when I went to make it.
What the negotiation actually looks like
The client asks with a ranked list:
GET /article HTTP/1.1 Host: example.com Accept: text/markdown, text/html;q=0.8
That reads as: I would prefer Markdown, I will take HTML at 80% preference. No q means q=1.
The server picks a representation, sets Content-Type to whatever it actually sent, and, critically, sets Vary: Accept so caches know the response depends on the request header:
HTTP/1.1 200 OK Content-Type: text/markdown; charset=utf-8 Vary: Accept
If it cannot satisfy any type the client will take, it returns 406 Not Acceptable.
Miss the Vary: Accept and your CDN will cheerfully serve the Markdown version to a browser and the HTML version to an agent, at random, for as long as the cache lives. That is the one mistake in this whole exercise that will visibly break your site, so it is worth stating twice.
Why not just ship .md sibling files
Because a sibling file at /article.md is a different URL. Your canonical URL keeps returning HTML to any agent that asks, your links and your ranking stay pointed at the HTML, and you now maintain two addresses for one piece of content.
Siblings are not wrong, they are just incomplete. If you ship them, advertise them with a Link: rel="alternate" header per RFC 8288 so agents can discover them, and understand that you have added a discovery mechanism, not solved negotiation. You can do both, and there is a decent argument for doing both.
The part that surprised me: Astro cannot do this on a static site
Astro's middleware is the natural home for Accept negotiation, and the official recipe is a middleware that parses the header properly, handles q values, respects q=0 rejections, and appends Vary: Accept to the response.
None of that runs on my site. With output: 'static', which is the default and which this blog uses, Astro prerenders every page at build time. Middleware runs once, during the build. The emitted HTML then gets served by nginx without Astro being involved again, so request-time negotiation at the Astro layer silently does nothing. Not an error, not a warning. It just never fires.
Your options from there are the ones you would guess. Switch to output: 'server' and run a Node process, which for a blog of static articles means adding a runtime to something that currently has none. Put Cloudflare in front and flip on its managed Markdown negotiation at the edge, origin unchanged. Or keep the static build, emit both .html and .md at build time, and let the web server pick.
For a 506-post blog served from one box, the third option is the only one that does not add a moving part.
The nginx version
The recipe is a map block plus try_files:
map $http_accept $preferred_ext {
default ".html";
"~*text/markdown" ".md";
}
server {
listen 443 ssl http2;
server_name solooperatorstack.com;
root /var/www/solooperatorstack;
add_header Vary Accept always;
location / {
try_files $uri$preferred_ext $uri/index$preferred_ext $uri.html $uri/index.html =404;
}
location ~* \.md$ {
default_type text/markdown;
charset utf-8;
add_header Vary Accept always;
}
}
Three things about this that are easy to get wrong. The second try_files entry is not redundant: without $uri/index$preferred_ext, a request for /about/ with a Markdown Accept header falls straight through to /about/index.html even when /about/index.md exists. The repeated add_header in the nested location is also not a copy-paste error, because add_header in an inner block replaces the outer one rather than adding to it. And nginx does not know what a .md file is, so without default_type you will serve Markdown labelled as something else.
The honest caveat, which the recipe states itself: map substring-matches the header. Accept: text/markdown;q=0, text/html will match the Markdown branch even though the client explicitly rejected Markdown. In practice no agent sends that, but it is not RFC-correct, and if that bothers you the answer is to proxy the Markdown branch to something that parses Accept properly.
For my site, emitting the .md alongside the HTML is the actual work. The content is already Markdoc in src/content/posts/<slug>/index.mdoc, so the source of truth exists. What is missing is a build step that writes the body out to dist/<slug>/index.md with the frontmatter stripped and the author-only comment blocks removed, which is maybe thirty lines against the content collection API.
Measure before you build any of this
Here is the step I would put ahead of all of it, and the reason this post is not just the config.
Grep your access logs for the header. If you are on nginx, $http_accept is not in the default log format, so you probably are not capturing it at all, which is the first thing to fix:
log_format withaccept '$remote_addr - $status "$request" "$http_user_agent" accept="$http_accept"';
Run that for a week. Count how many requests arrive with text/markdown anywhere in the Accept header, and which user agents send it. I expect that number to be small right now. Cloudflare's edge feature and this site's existence suggest it is going up, and the config is cheap enough that "small but rising" justifies it. But I would rather write "I saw N requests last week" in a follow-up than assert that the agentic web has arrived because a landing page told me so.
What I'd actually do
Add the accept= field to your access log format today. That is one line, it costs nothing, and in a week you have data instead of a vibe.
If you already run behind Cloudflare, flip on the edge feature and stop reading, because you get the whole thing for a toggle and no origin change.
If you self-host static, write the .md emitter first and ship the sibling files with a Link: rel="alternate" header. That is useful on its own, it is testable in isolation, and it is the prerequisite for the nginx negotiation anyway. Then add the map block once you have seen the header in your own logs.
Where this could be a waste of time
The case against: agents overwhelmingly do not send Accept: text/markdown today. They send a browser-ish Accept header, get HTML, and run it through a converter that has been fine at stripping nav and scripts for years. The token savings are real but they accrue to the agent operator, not to you. You are doing work so that somebody else's crawler is cheaper to run, and the theory that this buys you better retrieval or better citation treatment is, right now, a theory.
The stronger version of that objection is that this is the same shape as every "just add this file and the AI will love you" proposal of the last two years, most of which turned out to be cargo cult. llms.txt has not obviously moved anything for anyone I know.
Where I think content negotiation is genuinely different: it is not a new convention anyone has to agree to adopt. It is a mechanism already in HTTP, already implemented in every client and server, doing precisely the job it was designed for. That is a much better bet than a new file at a new well-known path. But it is still a bet, and the log line comes first.
Author
Lukas
@lukcombinator