Do I Need an llms.txt File for My Website? (Myth vs. Fact)

No, not urgently, and probably not at all unless a developer is already touching your site for other reasons. An llms.txt file (a proposed, robots.txt-like file meant to help AI systems understand a site) is nearly free to create, but 2026's actual crawler data shows AI systems mostly aren't reading the ones that already exist, and having one shows no measurable connection to getting cited more often. If you've read a guide or gotten a pitch recommending one, the honest, data-backed answer is: it's not harmful, it's just not the lever most of that advice implies it is.

This is a follow-up to our broader look at AI search optimization hype, which covers the AEO/GEO landscape generally. This piece goes deep on the one specific tactic that comes up most often: should you spend the time (or pay someone) to build this file.

What an llms.txt file actually is

It's a plain text file, usually placed at yoursite.com/llms.txt, listing a short summary of a business and a curated set of its most important pages with brief descriptions, written in a format an AI system could theoretically parse quickly. The idea, first proposed in September 2024, is that it gives AI models a cleaner shortcut than crawling and interpreting full HTML pages.

It doesn't replace a sitemap or robots.txt. A sitemap tells search engines what pages exist; robots.txt tells crawlers what they're allowed to access. llms.txt is a suggestion, not an instruction, and nothing is required to respect it.

What the 2026 data actually shows

This is where most advice on the topic stops short. Here's what's actually been measured:

• Adoption is climbing, but it's still a small minority. A study of 300,000 domains found roughly 1 in 10 sites (10.13%) had implemented llms.txt, with adoption spread fairly evenly across traffic tiers. Among the top 1,000 sites by traffic specifically, a separate tracker put adoption at 8.7% (rising to about 16% once unreachable infrastructure domains are excluded from the count).
• Growth is real in relative terms. One tracking service following 3 million-plus sites recorded llms.txt instances growing nearly 9x between June 2025 and May 2026. Still a small fraction of the sites tracked overall.
• AI systems mostly aren't reading the ones that exist. Server-log analysis across 137,000 domains found that 97% of llms.txt files received zero requests in May 2026. Of the requests that did happen, AI retrieval bots made up only about 1% of total traffic to those files. The largest single share of requests came from SEO audit tools and general web crawlers, not the AI systems the file is meant for.
• No measurable link to getting cited more. The same 300,000-domain study ran a statistical test and a machine-learning model to check whether having llms.txt predicted how often a domain got cited by AI systems. It found no meaningful correlation, and the model actually got more accurate once the llms.txt variable was removed, meaning it was adding noise, not a real signal.
• Google has said directly that it isn't necessary. Google's own guidance on optimizing for generative AI search features lists creating special AI markup or llms.txt files among the things not worth doing, since AI-generated search features draw on the same crawlable, publicly indexed content Search already uses. There's no separate AI crawler being courted with a special file.

Put together: this isn't a case of "the data's inconclusive." Multiple independent studies, using different methods (server logs, statistical correlation, machine learning), landed on the same answer.

Where it does get used

There's one place llms.txt sees genuine, working adoption: coding tools and AI-assisted development environments (the kind that reads a codebase or documentation to help write software) do fetch and use these files. That's a real, if narrow, use case, and it has nothing to do with a consumer AI system deciding whether to cite a business in an answer. Unless your website is itself developer-facing documentation, this use case doesn't apply.

So why does most advice still say "add it anyway"?

Because the argument "it costs almost nothing, so why not" is genuinely reasonable on its face, and most of what's written about llms.txt stops there without checking whether the "almost nothing" cost also comes with "almost nothing" benefit. It does. The honest version of "why not" isn't "it might help and can't hurt." It's "it currently does close to nothing, and the businesses telling you it's essential are usually the ones selling the work to build it."

That doesn't make it a bad idea in an absolute sense. If a developer is already touching your site for another reason and adding one takes them ten minutes, there's no real downside. What doesn't hold up is treating it as a meaningful AI-visibility strategy, paying someone specifically to build one, or believing a site without one is at a real disadvantage against a site with one.

What actually helps you show up in AI search results

If the goal is genuinely being found and cited by AI systems, the effort is better spent on the fundamentals that the same research keeps pointing back to: a crawlable, well-structured site, content that answers questions clearly and directly, real expertise behind what's written, and a solid technical foundation search engines can actually read. Our SEO basics checklist covers exactly this ground, and it's the same list AI visibility depends on, not a separate one.

---

Not sure whether your site actually has those fundamentals in place? That's exactly what our free Website Grader checks. Run it and we'll show you what's worth fixing first, no llms.txt file required.