llms.txt is a simple idea with an outsized amount of hype around it: a markdown file at the root of your site that gives large language models a curated map of your most important pages. Two years after the proposal, it is fair to ask the adult questions. Who actually adopted it? Is there evidence AI systems read it? And if you decide to publish one, what does a good file look like?

This article sticks to what can be verified, links to the original data, and is honest about what is still unknown. (If you want the beginner's introduction first, read What is llms.txt? — this piece is about the state of adoption and evidence in 2026.)

A 60-Second Recap

The standard was proposed in September 2024 by Jeremy Howard of Answer.AI (original proposal; full spec at llmstxt.org). The pitch: context windows are limited and HTML pages are cluttered with navigation, scripts, and ads, so give LLMs a clean, curated markdown index at /llms.txt — a title, a one-paragraph summary, and grouped lists of your key pages with one-line descriptions. An optional companion, llms-full.txt, inlines the full content.

Unlike robots.txt, which tells crawlers what they may not read, llms.txt tells AI systems what is most worth reading. One controls access; the other proposes priorities.

Adoption: Real Growth From a Small Base

The most consistent longitudinal numbers come from Originality.AI's public tracking study, which monitors llms.txt files across roughly three million websites. Their tracker recorded 4,088 llms.txt files in June 2025 and 36,120 by May 2026 — an 8.8× increase in twelve months (source).

Among popular sites, an independent May 2026 crawl of the Tranco top 10,000 domains found 586 valid llms.txt files — 5.86% adoption (source). The same study contains a detail worth pausing on: of all URLs that returned HTTP 200 for /llms.txt, about 64% failed validation — redirects to homepages, HTML error pages, empty files. Naively counting 200 responses overstates real adoption by nearly 3×, so treat any adoption headline without a validation step skeptically.

Who adopts skews heavily technical: developer-tool companies, documentation sites, and AI startups dominate the public directories (llmstxt.site, directory.llmstxt.cloud). Anthropic — the company behind Claude — publishes one for its own documentation at docs.anthropic.com/llms.txt.

The Evidence Question: Does Anyone Read It?

Here is the honest part. As of this writing, no major AI provider — OpenAI, Anthropic, Google, Perplexity — has publicly committed to consuming llms.txt in their production answer systems.

The most-quoted skeptic is Google's John Mueller, who wrote in April 2025: “AFAIK none of the AI services have said they're using llms.txt (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag” (via Search Engine Journal). He has since added a deeper argument: because llms.txt is self-reported by the site that wants to be chosen, an LLM cannot use it to differentiate between competing sites — every site would simply claim to be the best (source).

The counterpoint, argued by practitioners (for example at Search Engine Land), is that Mueller's comparison addresses ranking, not comprehension: llms.txt was never proposed as a ranking signal, but as a convenience for AI agents that fetch your site at answer time, and request logs — not announcements — are where its consumption would first show up. Both things can be true: the file may help an agent that already chose to read your site, while doing nothing to make agents choose you.

What we can say from our own vantage point at RankedByAI, having scanned many small-business sites: we have not seen convincing public evidence that publishing llms.txt by itself moves how often a business is mentioned in AI answers. If someone sells you llms.txt as a visibility silver bullet, hold onto your wallet.

So Should You Publish One?

Our honest position: yes, if it costs you ten minutes — and with expectations set correctly.

  • The cost is near zero. It is one static text file. There is no penalty, no crawl-budget impact, no downside beyond maintenance.
  • The realistic upside is comprehension, not ranking. If an AI agent fetches your site while answering a question, a clean curated index can only help it find your services, locations, and proof faster.
  • The speculative upside is optionality. If a major provider starts consuming the file, early adopters inherit the benefit without lifting a finger.

What llms.txt does not replace: crawlable pages (check your robots.txt with our AI crawler checker — a blocked crawler never sees your llms.txt either), clear service pages, reviews, and third-party mentions. Those remain the signals that demonstrably move AI recommendations.

How to Write a Good One

The format, per the spec, is deliberately minimal markdown:

  • An H1 with your site or business name — the only required element.
  • A blockquote immediately after: one or two sentences summarizing what you do, for whom, and where.
  • H2 sections grouping links — for a business, typically Services, Locations, Proof (reviews, case studies, credentials), and Contact.
  • Each link on its own line: [Page title](URL): one-line description.
  • An optional ## Optional section for secondary links an agent can skip when context is tight.

Practical rules that separate useful files from junk: keep it under a couple of kilobytes; describe pages factually rather than promotionally (remember Mueller's point — self-praise is unverifiable and therefore ignored); link your most decision-relevant pages, not every URL in your sitemap; and update it when your pages change, because a stale index is worse than none.

You can write the file by hand in any text editor, generate one from your live site with our free llms.txt generator, or use the open-source CLI (npx rankedbyai llms-txt yoursite.com, source on GitHub) which drafts one from your sitemap and homepage links.

FAQ

Does llms.txt improve my rankings in ChatGPT or Perplexity?

There is no public evidence that it does, and no major AI provider has announced support. Treat it as a low-cost comprehension aid for agents that already visit your site, not as a ranking lever. The things that measurably affect AI recommendations are crawlability, clear factual pages, reviews, and third-party mentions.

What is the difference between llms.txt and robots.txt?

robots.txt controls access: it tells crawlers which paths they may or may not fetch, and the major AI crawlers do respect it. llms.txt proposes priorities: it suggests which pages matter most and summarizes your site. One is enforced, the other is advisory — and currently unconfirmed as consumed.

Do I need llms-full.txt too?

Probably not. The full-content variant is mainly useful for documentation sites whose entire value is text an agent might want in one request. For a typical business site, a well-curated index file covers the realistic use case, and the adoption data shows most adopters make the same choice — in the May 2026 Tranco crawl, only about one in six sites with a valid llms.txt also published a valid llms-full.txt.

How often should I update it?

Whenever the linked pages change meaningfully — new services, new locations, renamed pages. A quarterly review is enough for most small businesses. If you generate the file with a tool, regenerate it after site updates and diff the result before publishing.