llms.txt: What the File Actually Is, and What It Is Not

Every few months a new file appears at the root of websites and everyone asks the same two questions: is it required, and does it help my ranking. For llms.txt the answers are no and no. It is worth writing anyway, and the specification explains why in about two paragraphs, which is roughly two paragraphs more than most articles about it quote.

The two problems it addresses

The first is a hard limit. As the specification puts it, large language models increasingly rely on website information, but face a critical limitation: context windows are too small to handle most websites in their entirety. A model answering a question about your product cannot read your site. It can read a slice of it, and something has to decide which slice.

The second is quality. Converting complex HTML pages with navigation, ads, and JavaScript into LLM-friendly plain text is both difficult and imprecise. Even the part that gets read arrives degraded, mixed with menus and cookie banners and whatever your framework injected.

llms.txt answers both at once by moving the decision to you. Instead of a model guessing which twenty pages of your site matter and parsing them badly, you hand it a short, clean, curated list. That is the entire idea.

Where it lives, and why the location matters

The file belongs at the root path /llms.txt of a website, or optionally in a subpath. A predictable address is the whole point: anything that wants it can find it without being told.

It is tempting to file it mentally next to robots.txt, and the comparison helps as long as you keep the difference in view. robots.txt is a restriction: it says what a crawler may not fetch. llms.txt is a recommendation: it says what is worth reading first. One is a fence, the other is a signpost. Publishing a signpost does not remove your fence, and it does not build one either.

The structure is smaller than you think

Most of the anxiety around this file comes from assuming it has a demanding schema. It does not.

An H1 with the name of the project or site. The specification notes that this is the only required section. Everything else is optional in the strict sense.

A blockquote with a short summary of the project, containing key information necessary for understanding the rest of the file. This is the sentence that tells a model what it is looking at before it reads a single link, and it is the part most worth rewriting twice.

Zero or more markdown sections delimited by H2 headers, containing file lists. Group by what a reader would be trying to do: documentation, guides, reference, pricing. The grouping carries meaning, so vague headings waste it.

The Optional section is a real instruction, not a label

One section name has defined behaviour, and it is the part people get backwards. The specification states that the "Optional" section has a special meaning: if it is included, the URLs provided there can be skipped if a shorter context is needed.

Read that as what it is, a triage instruction you are writing yourself. You are telling a model which of your links to drop first when it runs out of room. Putting your best explanatory pages under "Optional" because the heading sounds modest is the one mistake that actively harms you: you have marked your most useful content as the first thing to discard.

It is an inference file, not a training file

This distinction settles most of the objections raised about llms.txt, and the authors state it plainly: their expectation is that llms.txt will mainly be useful for inference, that is, at the time a user is seeking assistance, as opposed to for training.

So the scenario it serves is narrow and concrete. Someone asks an assistant a question. The assistant goes looking. Your file is what it finds first, and it decides what gets read in the seconds that follow. If your worry is instead about models training on your content, this file is the wrong instrument entirely, and robots.txt directives are the conversation to have.

What it does not do

It is a proposed convention. Nothing obliges any provider to read it, and no search engine has promised anything in exchange for publishing one. Anyone telling you it is a ranking factor is describing a hope, not a mechanism.

The honest case for writing one is different and, we think, stronger: the cost is close to zero, the downside is nil, and the exercise of deciding which twenty pages actually represent your business is worth doing even if nothing ever reads the result. Most teams discover, halfway through, that they cannot agree on the list. That is the useful part.

Frequently Asked Questions

What problem does llms.txt actually solve?

Two, and the specification states both. The first is size: large language models increasingly rely on website information, but face a critical limitation, namely that context windows are too small to handle most websites in their entirety. The second is noise: converting complex HTML pages with navigation, ads, and JavaScript into LLM-friendly plain text is both difficult and imprecise. So llms.txt is not a ranking file or a permissions file. It is a curated, already-clean map of what matters on your site.

Where does the file go?

At the root path /llms.txt of a website, or optionally in a subpath. That is the whole of the location rule. It sits alongside robots.txt in spirit, at a predictable address, but it does the opposite job: robots.txt says what a crawler may not fetch, while llms.txt says what is worth reading first.

What is the required structure?

Less than most people assume. The specification calls for an H1 with the name of the project or site, and says this is the only required section. After it comes a blockquote with a short summary of the project, containing key information necessary for understanding the rest of the file, then zero or more markdown sections delimited by H2 headers, containing file lists. Everything past the H1 is therefore optional in the strict sense, which is why a useful llms.txt can be written in twenty minutes.

What does the Optional section mean?

It is a special section name with a defined behaviour, not a label you invent. The specification is explicit: if it is included, the URLs provided there can be skipped if a shorter context is needed. It is your own statement of priority, telling a model which links to drop first when it runs short of room. Putting your most important pages there defeats the purpose.

Is llms.txt for training models on my content?

No, and the authors say so directly: their expectation is that llms.txt will mainly be useful for inference, that is, at the time a user is seeking assistance, as opposed to for training. It is a file for the moment someone asks a question and a model goes looking for an answer. If your concern is training, llms.txt is the wrong lever and robots.txt directives are the conversation to have instead.

Does Google use llms.txt?

There is no obligation on any provider to read it, and treating it as a ranking factor is a misreading of what it is. It is a proposed convention, adopted by those who choose to. That said, the cost of publishing one is close to zero and the downside is nil, which is a rare combination. Write it because it makes your content legible to whatever does read it, not because a search engine promised anything.

The statement of the context-window limitation, the description of HTML-to-text conversion as difficult and imprecise, the root-path location rule, the required H1 and its status as the only required section, the blockquote summary, the H2 file-list sections, the special meaning of the "Optional" section, and the expectation that the format serves inference rather than training, are all taken from the llms.txt specification, checked at the time of writing. It is a proposed convention and it may change.

ZAX Support for AI Visibility

We help teams make their content legible to AI applications: structured data, llms.txt, MCP servers exposing internal systems, and the measurement that tells you whether any of it is working.

Audit and scoping. A free 30-minute AI audit reviews what an assistant can currently find about you, and what it gets wrong.