llms.txt is a proposed convention: a plain text file at the root of your website, written in Markdown, that gives AI systems a curated summary of your most important content. The pitch is appealing. AI assistants work with limited context and messy HTML, so why not hand them a clean map? The question that matters, though, is not whether the idea is tidy. It is whether any AI system actually reads the file. Here is the honest answer, as the evidence stands today.
What an llms.txt file actually is
The proposal came out of the AI developer community in 2024. The file sits at yoursite.com/llms.txt and contains a short description of your site plus a structured list of links to your key pages, each with a one-line summary, all in Markdown so a language model can read it without parsing HTML. A companion idea, llms-full.txt, goes further and packs full page content into a single file for tools that want everything at once.
Adoption is real but lopsided. A number of documentation platforms and developer tools now generate the file automatically, because their users literally paste docs into AI assistants and the format genuinely suits that workflow. Across the wider web of business sites, take-up is thin, and that gap tells you something useful about who the file is actually for.
llms.txt vs robots.txt: different jobs
The name invites the comparison, but the two files do opposite jobs. robots.txt is a long-established standard that tells crawlers what they may not access, and the major AI crawlers publicly document that they honour it. llms.txt is an invitation rather than a control: it suggests what AI systems should read first, and no major AI provider has committed to following that suggestion. One file has decades of convention and enforcement behind it. The other is a proposal still waiting for takers. If your goal is managing which AI crawlers access your site, robots.txt and your server settings are where that actually happens, as part of ordinary technical SEO.
What the evidence says about llms.txt
Here is the measured picture. Google has said publicly that it does not use llms.txt, and its representatives have compared the file to the old keywords meta tag: a place where site owners describe themselves, which search systems learnt long ago not to lean on. No major AI provider has confirmed that its systems fetch and honour the file when composing answers. Server logs shared around the SEO community show occasional fetches of the file by various bots, but a bot downloading something is not evidence that it influenced anything a user ever saw. Until a provider says "we read this and it affects our answers", the file's benefit remains hypothetical.
It is worth being clear about why the debate stays noisy. The file is cheap to generate, easy to sell as a deliverable, and impossible to disprove quickly, which makes it attractive to vendors hunting for something new to invoice. None of that is evidence. When we evaluate any AI visibility tactic for clients, the question is always the same: has anyone shown, on the record, that it changes what an engine says? For this file, so far, nobody has.
Set against that, the cost of having one is close to zero. The file harms nothing, no system penalises it, and if a future assistant does begin reading it, early adopters lose nothing by being ready. No confirmed benefit, negligible cost: that combination makes the file a judgement call, not a controversy, and certainly not the emergency some vendors present it as.
The trap to avoid: treating llms.txt as the strategy. Visibility in AI search comes from being the clearest, most credible answer on the open web, not from a manifest file. If the file is polished and your content is not, the order is backwards.
Should you add an llms.txt file?
Our advice runs in two parts. If you run a documentation-heavy or developer-facing site, add one. Your audience already feeds your content into AI assistants, the format helps that workflow directly, and the effort is minutes. For everyone else, treat it as an optional extra at the end of the list, after the work with evidence behind it: content built the way AI engines actually cite, clean structure and schema markup, and the authority signals covered in our AI SEO explainer.
If you do add one, keep it truthful and current. List your genuinely important pages, describe them plainly, and regenerate the file whenever the site changes meaningfully. Generating it takes minutes with the free tools available, or a developer can script it from your sitemap so it refreshes itself. A stale llms.txt that contradicts your live pages is worse than none at all, because the one thing every AI system definitely reads is the content itself.
The wider discipline of earning citations in AI search, known as GEO, is where the real leverage sits, and it rewards substance over shortcuts. Our AI and GEO visibility service focuses on the signals that demonstrably move visibility, and our guide to what GEO is explains the strategy in plain English. That is also the order your budget should follow: content and structure first, authority second, experimental files last.
Want the priorities in order? Run the free AI visibility check to see whether AI engines cite your business yet. If the answer is no, this file is not the reason, and the results will show you what is.
