What Is llms.txt? And Does It Actually Work in 2026?
llms.txt is a proposed plain-text file you place at the root of your site to give large language models a clean, curated map of your most important content. The short answer to the question everyone is asking in 2026: it does not help you rank in Google, and the big AI search crawlers mostly ignore it — but it is quietly becoming useful in a different corner of the AI web. This guide separates the hype from what the file actually does.
The idea has been circulating since Jeremy Howard of Answer.AI proposed it in September 2024. Two years and a lot of blog posts later, marketers are still confused about whether to spend an afternoon adding one. So let's look at the evidence — Google's on-record position, real crawler-log data, and the one place llms.txt is genuinely earning its keep.
What is llms.txt, exactly?
llms.txt is a Markdown file served at https://yourdomain.com/llms.txt. The proposal is loosely modelled on robots.txt and sitemap.xml, but it solves a different problem. Where robots.txt tells crawlers what they may not touch, llms.txt tries to tell a language model what matters most and where to find the clean version of it.
A typical file is short and human-readable: an H1 with your site or product name, a blockquote summary, then curated sections of links — each with a title and a one-line description — pointing to your key docs, guides or pages. There is a companion convention, llms-full.txt, which inlines the full text of that content into a single file so a model can ingest everything in one fetch without crawling dozens of URLs.
The pitch is reasonable on paper. Modern web pages are bloated with navigation, cookie banners, scripts and ads. A model that has to parse all of that wastes context window and risks grabbing the wrong passage. A curated Markdown file hands it the signal without the noise.
Does llms.txt work in 2026? The honest answer
Here is where the marketing promise and reality part ways. As of mid-2026, the evidence is blunt on two points and encouraging on a third.
1. Google Search does not use it — full stop
This is not a rumour or an inference. Google has said no on the record, repeatedly. In 2025, Google's Gary Illyes confirmed Google does not support llms.txt and has no plans to, and John Mueller publicly compared it to the long-discredited keywords meta tag. Google's own Search Central documentation, updated in 2026, is unambiguous: Google Search does not use llms.txt for rankings, does not use it for AI Overviews, and does not use it at all. If your goal is to appear in Google's AI answers, this file will not move the needle.
2. The major AI search crawlers rarely fetch it
Server-log studies tell the same story from the other direction. The crawlers behind AI answer engines — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended — overwhelmingly skip /llms.txt and crawl your rendered HTML directly. No major model provider (OpenAI, Anthropic, Google, Meta or Mistral) has publicly committed to using llms.txt as a ranking or citation signal in its production search surfaces. The file can sit on your server for months and never get a single hit from the bots it was designed for.
3. But it is quietly working in the agentic layer
The twist is that the same file doing nothing for ChatGPT search citations is doing real work somewhere else: developer and agent tooling. IDE and coding agents — Cursor, Windsurf, GitHub Copilot, Claude Code, Cline and Aider — routinely look for /llms.txt and /llms-full.txt when you point them at a documentation site. This is the "business-to-agent" layer, where an AI assistant is actively working on a user's behalf and wants a compact, authoritative version of your docs. If you publish a developer product, API or technical library, llms.txt has a genuine, measurable audience.
Adoption: who is actually using it
Despite two years of industry conversation, uptake is modest. A widely cited 2026 study of roughly 300,000 domains found an adoption rate of about 10% — one site in ten — and growth has been slow rather than viral. The early adopters skew heavily toward developer-tooling and SaaS documentation sites, which lines up neatly with where the file is actually consumed. General marketing and e-commerce sites have largely added it on faith, hoping for an AI-search benefit that the data does not yet support.
That gap between hope and evidence is the single most important thing to understand before you invest time here. Adding llms.txt because a competitor did, or because a vendor promised "AI visibility," is optimising for a channel that mostly does not read the file.
llms.txt vs robots.txt vs sitemap.xml
These three files are easy to confuse because they all live at your site root and all speak to machines. They do very different jobs.
| File | Audience | Purpose | Respected today? |
|---|---|---|---|
robots.txt | All crawlers | Control what may be crawled or blocked | Yes — universal standard |
sitemap.xml | Search engines | List all indexable URLs for discovery | Yes — universal standard |
llms.txt | LLMs & AI agents | Curate the most important content in clean Markdown | Partly — agents yes, search crawlers mostly no |
One practical implication: if you genuinely want to control how AI crawlers treat your site today, robots.txt and your CDN rules do far more than llms.txt. That is the layer where publishers are actually fighting AI bots — see our breakdown of how Cloudflare now blocks AI crawlers by default for what that battle looks like in practice.
Should you add an llms.txt file? A decision guide
The cost of publishing one is low, and it will not hurt your SEO. But "it can't hurt" is a weak reason to spend time on anything. Decide by who you are:
- You run a developer tool, API, SaaS or documentation site → Yes, add it. This is the audience that actually fetches it. A clean llms.txt and llms-full.txt make your docs easier for coding agents to use, which is a real distribution channel in 2026.
- You publish a technical blog or knowledge base with many how-to pages → Reasonable to add. Some agentic tools will use it, and maintaining a curated content map is good hygiene regardless.
- You run a general marketing site, local business or store and want Google AI visibility → Skip it, or treat it as strictly optional. Google will not read it, and your effort is far better spent on the fundamentals that AI search actually rewards.
If AI-search citations are your real goal, the levers that work are the ordinary ones: authoritative, clearly structured, fact-rich content that both people and machines can parse. We cover that playbook in depth in our guide to optimising your content for AI search, and in how to optimise for Google's Information Agents — both of which will do more for your visibility than any root-level text file.
How to write an llms.txt file (if you decide to)
If you fall into the "yes" camp, keep it simple. The format is deliberately lightweight:
- Start with an H1 containing your site or product name — this is the only required element.
- Add a blockquote summary of one or two sentences describing what the site or product is.
- Group your key links into H2 sections such as "Docs," "Guides" or "API reference." Under each, list links as
[Title](url): one-line description. - Keep an optional "Optional" section at the end for links an agent can skip if it is short on context.
- Consider an llms-full.txt that inlines the actual Markdown content of those pages, so an agent can load everything in a single request.
- Keep it current. A stale content map is worse than none — update it when your key pages change, and automate its generation from your docs if you can.
Serve the file as text/plain or text/markdown at your root, and you are done. There is no submission step and no registry — agents that support the convention will simply try the URL.
The takeaway
llms.txt is a genuinely good idea aimed at a genuinely real problem, but in 2026 it is not the AI-search silver bullet many articles imply. Google does not use it, the big AI search crawlers mostly ignore it, and adoption sits around one site in ten. Its real, growing value is in the agentic layer — coding assistants and AI tools that consume documentation directly. So add it if you run a developer or docs-heavy product, treat it as optional everywhere else, and never let it distract from the structured, authoritative content that actually earns AI citations. The file is a bet on where the AI web is going, not a fix for where it is today.
