What Is llms.txt, Really? A Straight Answer

What it actually is

Think of it as a table of contents written for machines instead of people. A normal web page is wrapped in navigation, consent prompts, ads and markup that a model has to wade through. This file strips that away. You write a short Markdown document that names your project, describes it in a sentence, and links to the pages you most want a model to read, each with a one-line note on what it covers.

It is a proposal, not a law and not a search engine feature. Jeremy Howard of Answer.AI published the idea in September 2024, and the spec lives at llmstxt.org. The problem it tries to solve is genuine: model context windows are too small to swallow an entire site, so the document offers a clean, pre-digested shortlist a model can pull in at the moment it answers a question.

What actually goes in it

The format is deliberately plain. The spec asks for a specific order, and only the first line is strictly required.

• An H1 with the name of your site or project. This is the only mandatory part.
• A blockquote with a short summary of what the site is and who it is for.
• Optional paragraphs giving extra context a model should know before it reads on.
• One or more H2 sections, each a list of Markdown links to important pages, with a short description after each link.
• An optional section literally headed Optional, holding secondary links a model can skip when it is short on room.

Many sites pair it with a second document called llms-full.txt, which concatenates the full text of the linked pages into one long file. The short one is the index. The long one is the whole library flattened into Markdown. Tooling like Mintlify generates both automatically for the docs sites it hosts, which is a large part of why the format spread at all.

Who reads it, and who does not

This is where the honest answer diverges from the hype, so I want to be precise. There is a difference between a file existing and a machine choosing to fetch it.

As of 2026, no major AI provider has publicly committed to reading third-party llms.txt files when its crawler visits your site. Google has been the most direct about this. Its Search Advocate John Mueller compared the format to the old keywords meta tag: the tag search engines eventually abandoned because it was just a site owner claiming what a site was about, with nothing verifying it.

AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it).

Google has separately said its Search systems, including its AI features, do not use the file to rank or surface pages. So if your reason for adding one is to climb Google or to show up in AI overviews, the public evidence does not support that today. The place it genuinely helps is narrower: when a person or a tool hands the URL directly to a model. Paste the file into a chat, point a coding assistant at it, or feed it to an answer engine that accepts a source link, and the model gets a clean map instead of a scraped mess.

Who publishes one anyway

Plenty of serious companies ship one, which muddies the picture, so it is worth separating two things. Hosting the file on your own documentation is useful even if no crawler auto-reads it, because your users and their coding assistants can point at it on purpose.

Anthropic's developer docs publish one, and I checked before writing this: their docs expose both the short index and a full version at the documentation root. Mintlify rolled out automatic generation across the docs sites it hosts in late 2024, which is how thousands of documentation sites, from AI labs to developer tools, ended up carrying the file almost overnight. Third-party trackers now count hundreds of thousands of sites with one. Adoption of publishing it is real. Adoption of reading it is the part still missing.

Should you add one to your site

Here is how I would decide, as someone who has to make this call for my own product. If you run documentation or an API that developers feed into coding assistants, add one. The payoff is concrete: people deliberately point tools at your docs, and a clean Markdown map genuinely helps them. Many doc platforms generate it for free, so the cost is close to zero.

If you run a small marketing site or a blog with a few hundred visitors a month and your goal is more search traffic, I would not lose sleep over it. It will not hurt you, and it costs little to add, but treating it as a search lever is chasing a signal nobody has confirmed exists. Ship your content, keep it readable, and revisit this the moment a provider you care about actually announces support.

The one thing I would not do is pay an agency to build and maintain one on the promise that it lifts your rankings. That is selling a certainty the evidence does not currently back.

How I think about it for measurement

My actual interest here sits downstream. Whether or not you add the file, the real question for a founder is whether AI answer engines are sending you anyone, and most analytics setups are bad at showing it. Traffic from an AI tool often arrives with no referrer, or a referrer that gets lumped into direct, so it hides in plain sight.

I build Conclick, so treat this as disclosure, but the honest job here is measurement, not a manifest file. Before you optimise for machines, get to the point where you can see which sources, including AI ones, actually bring people who then do something on your site. Otherwise you are tuning for a channel you cannot even confirm is running.