llms.txt Explained: What the Data Says and Who Needs One

llms.txt is a plain text file at the root of a website, written in Markdown, that tells AI tools what the site is and which pages matter. It sits next to robots.txt and sitemap.xml. It was proposed in 2024 and revised in 2026, about one in ten sites now publish one, and the best evidence so far says AI crawlers rarely read it and it does not change how often a site is cited. It is still worth understanding, because it is cheap, it clearly helps one kind of site, and a stale one can do harm.

This post covers what the file is, what the data says, where it genuinely helps, and how to decide whether your site needs one. It is part of our guide to agent-ready websites.

Where it came from

Jeremy Howard, co-founder of Answer.AI and fast.ai, published the llms.txt proposal in September 2024. His problem was practical: language models have limited context windows, and a website's HTML is full of navigation, scripts and layout that waste that space. A short Markdown file listing the important pages, ideally with clean Markdown versions of each, would let a model find what it needs quickly.

In August 2026 he published a second version, revised after two years of adoption. The core idea is unchanged. What changed is mostly about how tools find the file and the clean page copies, covered below.

Notice what the proposal is for. Its main use case is a model reading documentation at the moment a person asks a question, for example a developer asking a coding assistant how a library works. It was not designed as a ranking signal for search.

What the file looks like

The proposal defines a fixed shape:

An H1 with the name of the site or project. This is the only required part.
A blockquote with a one-line summary.
Optional paragraphs of context.
Sections under H2 headings, each a list of links with a short note.
An optional section called "Optional", for links a tool can skip when space is tight.

A small business might publish something like this:

# Kaya Dental Clinic

> Family and cosmetic dentistry in Izmir, Turkey. English and Turkish spoken.

Kaya Dental Clinic has treated patients in Karşıyaka since 2009.

## Services
- [Implants](https://example.com/implants): Single and full-arch implants, prices and process
- [Whitening](https://example.com/whitening): In-clinic and take-home options

## About
- [Our dentists](https://example.com/team): Qualifications and languages
- [Contact](https://example.com/contact): Address, hours, booking

## Optional
- [Blog](https://example.com/blog): Patient guides

The proposal also suggests two companions. llms-full.txt puts the full text of the important pages into one file, so a tool can load everything at once. Markdown copies of pages, served at the same address with .md appended, give a clean version of each page. Both are common on documentation sites and rare elsewhere.

The 2026 revision adds two things worth knowing:

Discovery from the page itself. A page can point to its Markdown copy with <link rel="alternate" type="text/markdown"> and to the llms.txt that covers it with rel="describedby", either in the HTML or in an HTTP Link header. A tool no longer has to guess where the files are.
Files for part of a site. An llms.txt no longer has to sit at the root. A file at /docs/llms.txt covers the pages under /docs/, and when several apply, the most specific one wins.

What the data says

Three kinds of evidence are available, and they point the same way.

Adoption is real but modest. SE Ranking's analysis of roughly 300,000 domains found an llms.txt file on about 10% of them, with adoption nearly the same across low, medium and high traffic sites.

Crawlers rarely ask for it. OtterlyAI ran a 90-day server-log experiment, published in early 2026 and summarized by Search Engine Land. Of more than 62,100 AI bot requests to its test site, 84 went to /llms.txt, about 0.1%. An average content page on the same site drew around three times as many bot visits as the file did. Smaller log studies by other site owners report similar results, often zero requests.

It does not correlate with citations. The same SE Ranking study tested whether sites with the file are cited more often in AI answers. It found no statistically significant relationship. Removing llms.txt from their prediction model made the model slightly more accurate, not less.

The search companies have said so. Google's guidance on AI features and your website says no new machine-readable files are needed to appear in AI Overviews or AI Mode. Google's John Mueller has said publicly that no AI system has claimed to use llms.txt. None of OpenAI, Anthropic or Perplexity lists it in their crawler documentation as something their bots read.

Taken together: for a typical business site, llms.txt is not what decides whether an assistant finds or cites you. The 2026 revision makes the file easier to find; it does not change who chooses to read it.

Where it does help

The file earns its place in one clear case: documentation that people load into AI tools on purpose. A developer who wants a coding assistant to understand a product can point it at the product's llms.txt or llms-full.txt and get clean, complete context in one step.

That is why Anthropic, Cloudflare, Stripe and many open-source projects publish one for their developer documentation, and why documentation platforms such as Mintlify generate it by default.

There is a smaller second case. When a person pastes your site into an assistant and asks it to explain your business, a user-triggered fetcher may try the file if it exists. That is a handful of requests, not a traffic source, but the file is the one place where your own summary of yourself is read without layout getting in the way.

What it cannot do

It cannot make an assistant cite you. The evidence above says presence and citations are unrelated.
It cannot replace the page. An assistant that cites you reads the page it cites. If that page is unclear, a summary elsewhere does not fix it.
It cannot override robots.txt. It is not an access control file. If a bot is blocked, the summary does not let it in, and publishing it does not invite anything in either. That decision is covered in Should You Block AI Crawlers?
It cannot stay accurate on its own. A hand-written file is right on the day it is written and starts drifting the first time a page changes.

The drift problem

That last point decides whether the file is worth having. I have watched this happen with every hand-maintained SEO file over 25 years: the sitemap that lists deleted pages, the schema with an old address, the redirects nobody updated after the second redesign.

llms.txt has the same failure mode with a worse outcome. A stale sitemap wastes a crawler's time. A stale llms.txt tells a model something false about your business, in your own words, with your name on it: a service you dropped, a price you changed, an address you left.

Should your site have one?

A simple way to decide:

You publish technical documentation or an API. Yes, and consider llms-full.txt too. This is the use it was designed for.
Your platform generates it from the site automatically. Keep it. It costs nothing and cannot drift.
You would have to write and maintain it by hand. Spend the time on your pages and your structured data instead. If you do write one, put a reminder next to every content change.
You were told it will get you into ChatGPT answers. It will not. Crawl access, clear pages and consistent business details matter far more. The entity schema post covers the part most small sites are missing.

How to check yours

Open yourdomain.com/llms.txt. If you get a page instead of plain text, the file does not exist and your site is returning a fallback page.
Check every link in it still works and every description is still true.
Compare the business name and description against your homepage and your structured data. They should match word for word.
Search your server logs for requests to /llms.txt to see whether anything reads it at all.

How this works in Beste

Every published Beste site has an /llms.txt generated from the site itself: the site description, the entity details, every page with its localized URL in each language, every post and the profile links. When a page is added, renamed or removed, the file changes on the next publish. There is nothing to switch on, and SEO settings let you edit it if you want to.

We publish one for the reasons above, not because it gets anyone cited. It is free when generated, it cannot drift, and it gives the few tools that read it the same facts as your pages and structured data. You can see ours at beste.co/llms.txt. The agent-ready websites page lists everything else Beste publishes alongside it.

Sources

Checked in October 2026.