Every few months the SEO world gets a new file to argue about. This one is llms.txt. Here is the honest version: it is cheap to add, no major model is confirmed to use it, and your crawler settings matter far more.
What is llms.txt?
llms.txt is a plain-text file you place at yourdomain.com/llms.txt. It lists your most important pages, with short descriptions, in clean Markdown, so an AI tool can find the good stuff without wading through your navigation and cookie banners. Some sites also publish llms-full.txt with the actual content inlined.
It is a community proposal, not an official standard. Google has said it is not using it. OpenAI and Anthropic have not confirmed using it for ranking.
What controls AI access
Your robots.txt rules for the AI crawlers do the real work:
| Bot | Purpose | Common choice |
|---|---|---|
Googlebot | Search and AI Overviews | Always allow |
Google-Extended | Gemini grounding and training | Allow for AI visibility |
GPTBot / OAI-SearchBot | ChatGPT training and search | Allow for ChatGPT citations |
PerplexityBot | Perplexity index | Allow for Perplexity citations |
Should you add one?
- If your CMS can generate it automatically, add it. Low effort, no downside
- Do not hand-maintain it for a large site. The cost outweighs any theoretical benefit
- Do not expect a ranking or citation lift from it
- Spend the time on robots.txt, page speed and quotable content instead
Bottom line: llms.txt is a nice-to-have that costs almost nothing if automated. It is not a growth tactic. Anyone selling it as one is ahead of the evidence.
Where it stands right now
Mahdi Hassan · Web Developer & Site Speed
If you want it, wire it into the build so it regenerates from your sitemap and page metadata. A hand-kept llms.txt goes stale in a month, and a stale one is worse than none.
Illustrative example
A documentation-heavy SaaS
Say a team adds llms.txt and llms-full.txt for their docs, generated from the existing content pipeline, then watches AI citations and assistant referrals for a quarter.
The realistic outcome today is no measurable change in AI visibility from the files themselves. Any lift that quarter comes from restructuring the docs answer-first, which is the work worth doing first, every time.