llms.txt: What It Is, Whether It Works, and How to Write One (Honest 2026 Guide)

The short version: llms.txt is a plain-text file that gives AI systems a curated map of your site. Adoption has grown sharply, yet a 2026 crawl-log study found that 97% of valid files in its dataset received no requests at all. Google says Search ignores the file, and no major AI platform has committed to using it as a visibility signal. We publish one anyway. The cost is 20 minutes, the downside is close to zero, and the upside, if the standard lands, is being early. Just do not let anyone sell it to you as an AI visibility strategy. It is a lottery ticket, not a plan.

Most guides on this topic are written by people selling AI visibility services who need llms.txt to matter. Clunky sells hands-on website improvement work and we are still telling you: the evidence is thin. Here is all of it, then the how-to.

What is llms.txt?

llms.txt is a proposed standard: a Markdown file at your site root (yourdomain.com/llms.txt) that tells large language models what your site is about and which pages matter most. Think of it as a curated reading list for machines, distinct from two files it is often confused with:

• robots.txt controls which crawlers may access which pages. It is about permission.
• sitemap.xml lists pages for search engines. It is about discovery and completeness.
• llms.txt highlights your most important content with context. It is about curation.

The proposal came from Jeremy Howard of Answer.AI in September 2024, aimed originally at technical documentation, where it made immediate sense: docs sites are exactly what developers ask AI assistants about, and a curated index can be easier to use than thousands of navigation-heavy pages.

Does llms.txt actually work? The evidence

Adoption is real. Usage is not. As of June 2026:

• Ahrefs analysed 137,210 domains using its Web Analytics product. About 38,000 published a valid llms.txt file, but 97% received zero requests for it during May 2026.
• Among the small minority of files that were fetched, only 1.1% of requests came from AI retrieval bots. Audit tools, general crawlers and tools studying llms.txt generated far more traffic.
• Google says Search does not use llms.txt and that publishing one will neither help nor hurt visibility in Google Search, including its generative features.
• OpenAI, Perplexity and Anthropic direct site owners to robots.txt and their named crawler documentation when discussing search visibility. None commits to llms.txt as a ranking or citation signal.

So the honest status in mid-2026: a grassroots standard with real momentum among publishers and no confirmed visibility benefit from the major AI platforms. The people writing these files vastly outnumber the machines reading them.

Then why publish one at all?

1. Asymmetric bet. Twenty minutes of effort against the possibility that a future retrieval system adopts the standard.
2. Documentation sites are the exception. Docs platforms can generate llms.txt automatically and coding assistants can be pointed at the file directly. This is where the standard already functions as a deliberate index.
3. It forces a useful exercise. Writing one makes you decide, in one page, what your site is for and which ten URLs matter most. Many founders discover they cannot answer that. The file is a symptom check.
4. Some tools do fetch it. Coding agents, SEO platforms and smaller agents request it when linked or explicitly directed. Marginal, but not nothing.

What llms.txt will not do: improve your Google rankings, guarantee ChatGPT citations this quarter, or substitute for the unglamorous work that actually drives AI visibility. That work is covered in how to get cited by ChatGPT and AI Overviews.

How to write an llms.txt file

The format is Markdown with a loose specification: an H1 with your site name, a blockquote summary, then sections of links with one-line descriptions.

# Clunky AI

> Clunky AI scans AI-built websites for the problems their builders
> ship by default: rendering crawlers cannot read, duplicate metadata,
> accessibility failures and weak trust signals.

## Core

- [Methodology](https://clunky.ai/methodology): how the Website
  Readiness Score is calculated
- [Free scan](https://clunky.ai/free-scan): a 90-second website
  readiness diagnostic

## Guides

- [Lovable SEO guide](https://clunky.ai/blog/lovable-seo-guide):
  fixing the SEO problems Lovable sites often ship with
- [AI visibility guide](https://clunky.ai/blog/get-cited-by-chatgpt-ai-overviews):
  practical steps for crawlability, citations and measurement

## Optional

- [Editorial policy](https://clunky.ai/editorial-policy): how Clunky
  tests claims and separates evidence from opinion

Rules that matter:

• Curate hard. Ten to twenty links. If everything is important, nothing is; that is what sitemap.xml is for.
• Descriptions do the work. Each link gets one line saying what a machine would learn by reading it.
• Use absolute URLs, UTF-8, no HTML, and serve the file as text/plain or text/markdown at the root.
• Use the ## Optional section for content that can be skipped when a model is short on context.
• Keep it true. Update it when your key pages change. A stale map is worse than none.

There is also llms-full.txt, which inlines fuller site context rather than only linking to pages. Unless you run documentation or have a deliberate agent workflow, skip it; maintenance cost can easily outstrip the current benefit.

How to check an llms.txt file

Fetch yourdomain.com/llms.txt directly in a browser: it should load as plain text, not redirect to your homepage or return a 404. Check that the first heading names the organisation, every link is absolute and live, and every description says something useful.

Then check your server logs a month later for requests to the path. That tells you whether anything is fetching it, which is data most publishers never inspect. Ours is live at clunky.ai/llms.txt, doing exactly what this article describes.

FAQ

What is llms.txt in simple terms?

A plain-text file at yourdomain.com/llms.txt that gives AI systems a short, curated summary of your site: what it is, and the handful of pages that matter most, each with a one-line description.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers what they may access; llms.txt suggests what they should read first. One is a set of access rules, the other is a reading list. AI crawler permissions still belong in robots.txt.

Do ChatGPT, Claude or Google actually read llms.txt?

There is no confirmed visibility benefit from the major platforms today. Google explicitly says Search ignores llms.txt, while OpenAI, Anthropic and Perplexity direct publishers to their robots.txt and crawler controls. Some coding agents and smaller tools fetch the file when directed to it.

Should I create an llms.txt file?

If it takes you under half an hour: yes, as a cheap bet with little downside. If anyone proposes charging meaningful money to create or “optimise” one, decline; there is no mature platform behaviour to optimise against yet.

Does llms.txt help SEO?

No. Google says directly that the file has no effect on Search visibility, positive or negative. Its plausible value is as a convenient index for agents or tools that deliberately choose to use it.

References

• The original llms.txt proposal
• Ahrefs: analysis of llms.txt requests across 137,210 domains
• Google Search Central: generative AI search guidance
• OpenAI: publisher and crawler guidance
• Perplexity crawler documentation
• Anthropic crawler documentation

---

Ready to make the file? Generate or check your llms.txt free. To test whether crawlers can actually read the wider site, run a free scan, or ask about the Clunk Removal Sprint for hands-on fixes.