llms.txt: What It Is and Whether It Does Anything Yet

Type "llms.txt" into any search bar in 2026 and the results split cleanly into two camps: guides insisting every site needs one, and studies showing it does not do what those guides claim. Here is what the file actually is, and what the best data available says about whether it works.

What llms.txt actually is

llms.txt is a proposed markdown file, conventionally placed at a site's root (/llms.txt), that lists a website's most important pages in a short, structured format designed for a language model to read directly. The idea, per the current spec at llmstxt.org, is a simple template: an H1 with the project or site name, a one-line blockquote summary, optional context paragraphs, and then H2-headed sections of markdown links pointing to the pages worth reading, each with a short description.

The problem it is trying to solve is real. A model's context window is limited, and a full HTML page is full of navigation, ads, and script noise that has nothing to do with the content a model actually needs. Jeremy Howard, co-founder of Answer.AI and fast.ai, published the original proposal in September 2024, framing it as a sitemap-like convenience: instead of a crawler reconstructing meaning out of a rendered page, the site hands over a clean, curated index directly. A version 2 of the spec, reflecting close to two years of real-world adoption experience, was published in August 2026.

It is worth being precise about what llms.txt is not. It is not robots.txt. robots.txt is a permissions file, it tells a crawler what it is and is not allowed to fetch. llms.txt is a content index, a curated table of contents aimed at a model that has already decided to read the site. Confusing the two leads to the common mistake of assuming an llms.txt file changes crawl access, when it changes nothing about whether a bot can reach a page in the first place.

What a well-formed llms.txt looks like

The spec is deliberately small, which is part of its appeal. A minimal, compliant file looks something like this:

# Yabado

> Yabado runs generative engine optimization (GEO) audits and services, helping
> brands understand and improve how they are cited by ChatGPT, Perplexity, and
> Google AI Overviews.

## Docs

- [GEO Audit](https://yabado.ai/geo-audit): what a GEO audit measures and how to read the results.
- [GEO Services](https://yabado.ai/geo-services): ongoing GEO work, explained.

## Zine

- [GEO vs SEO](https://yabado.ai/zine/geo-vs-seo): how the two disciplines differ in practice.

That is the entire format: a name, a one-line summary, and grouped links with short descriptions. There is no metadata field for keywords, no ranking signal, nothing to game beyond writing an accurate description. That simplicity is exactly why it spread quickly among developer tools and documentation sites, where a clean, current index of docs pages is useful to write once regardless of whether any AI engine reads it.

Real examples exist, and some are widely used

Adoption is real, if uneven. Anthropic maintains its own llms.txt and llms-full.txt files for its documentation at docs.anthropic.com/llms.txt and platform.claude.com/llms-full.txt, which Anthropic's Alex Albert announced on X in November 2024 as a way to get the full documentation set into any LLM's context in one file. Documentation platforms including Mintlify generate an llms.txt automatically for every site they host, which is a large share of the adoption number below by itself. That is the honest state of things: it is a real, working convention that plenty of technical documentation sites have adopted for the same reason they adopt a sitemap, not evidence that AI answer engines factor it into who gets cited.

What the adoption and citation data actually shows

This is the part most llms.txt guides skip, because the answer is not flattering to the file.

SE Ranking analyzed roughly 300,000 domains in a study published in November 2025 and found llms.txt implemented on 10.13% of them overall. Adoption did not track site size in the way you might expect: sites with low traffic (0-100 monthly visits) had 9.88% adoption, mid-traffic sites (1,001-5,000 visits) had 10.54%, and high-traffic sites (100,001+ visits) actually had the lowest rate at 8.27%, according to Search Engine Journal's summary of the study. In plain terms: roughly one in ten sites has the file, and having more traffic does not make a site more likely to have added it.

The more important finding is about citations, not adoption. SE Ranking ran both a statistical correlation test and an XGBoost machine learning model to see whether llms.txt presence predicted which domains got cited by AI engines. The result: removing the llms.txt variable from the model actually improved its prediction accuracy. In the study's own framing, the file's presence "was adding noise, not signal" to citation behavior. There was no measurable correlation between having an llms.txt file and being cited more often. The same reporting notes that server log analysis shows AI crawlers like GPTBot occasionally fetch the file, but rarely, and with no observed tie to citation outcomes.

That single result is worth sitting with. This is not a case of "we don't know yet." It is a large, direct test that found no effect, run by a company that had every commercial incentive to find one.

What the AI companies themselves have actually said

Separate from the third-party study, it is worth checking what the AI labs have said in their own words, because the honest answer is: mostly, they have said no.

Google has been the most direct. At Search Central Live's Asia Pacific event on July 23, 2025, Google's Gary Illyes stated plainly that Google does not support llms.txt and is not planning to, according to Search Engine Land's report on the session. His guidance for showing up in AI Overviews was blunter still: use normal SEO practices, nothing generative-engine-specific required.

OpenAI's public crawler documentation covers OAI-SearchBot, GPTBot, and ChatGPT-User in detail, including robots.txt controls for each, but does not ask site owners to create or maintain an llms.txt file as part of its guidance.

Anthropic is the closest thing to a counterexample, and even that is limited. It maintains its own llms.txt and llms-full.txt files for its documentation, which is a real, checkable fact. What has not happened is any AI lab, Anthropic included, confirming that a page's llms.txt entry changes whether Claude, ChatGPT, or any other model actually cites it. Publishing a well-formed llms.txt for your own docs and using it as a citation-ranking signal for the rest of the web are two different claims, and only the first one is currently backed by anything a lab has said on the record.

There is one more data point worth naming honestly because it is anecdotal rather than a study: individual site owners have reported in their own server logs that OpenAI's crawlers request their llms.txt file periodically. That is a real observation from real logs, but it is a single site's anecdote, not a controlled test, and it says nothing about whether the fetch influenced any citation decision. It sits alongside, not against, SE Ranking's controlled finding of no measurable citation effect across 300,000 domains.

Why the hype outran the evidence

It is worth asking why a file with this little proven effect became a fixture of every GEO checklist published in 2025. Two things happened at once, and they got conflated. First, the underlying problem llms.txt describes is genuine: model context windows are limited, and a full rendered HTML page wastes a meaningful share of that budget on navigation and script noise that a model has to parse and then discard. Second, "solves a real technical problem for models processing your site" quietly became "therefore improves whether you get cited," and that second claim never had evidence behind it. It only had the intuitive appeal of the first claim borrowed for a different job. The SE Ranking study is useful precisely because it tested the second claim directly, rather than the first.

The practical takeaway

Put the study and the vendor statements together and the picture is consistent rather than contradictory. llms.txt is inexpensive to create, genuinely used by a meaningful slice of technical documentation sites, and, as of the best available evidence, not a lever that changes who gets cited by an AI answer. Google has said it will not use it. OpenAI's documentation does not ask for it. Anthropic uses one for its own docs without claiming it affects citation for anyone else's. And a 300,000-domain test found the file added noise to a citation-prediction model rather than signal.

None of that means skip it entirely. It costs an afternoon, it cannot hurt, and it is the kind of low-risk hygiene worth doing once and forgetting, the same category as a clean XML sitemap. What it should not do is take priority over the two things the evidence does support: making sure AI crawlers can actually reach and read your pages, and writing passages that state a complete, checkable fact on their own, which is the actual difference between GEO and ordinary SEO. See GEO vs SEO: What Actually Changes for that distinction in full, and What Is Generative Engine Optimization? for the research behind what does move the needle on AI citations. If you want to know specifically where your own site stands on crawler access and citation, before spending time on files with no proven effect, a GEO audit checks exactly that.