Google has stated that it does not use llms.txt, and its own AI optimisation guidance names the file as unnecessary for AI Overviews and AI Mode. John Mueller has compared it to the keywords meta tag — a file people publish, that nothing reads.
The file is not worthless. It is just not a search technique, and almost every article recommending it conflates the two.
The decision that actually affects you is a different one: which AI crawlers you allow. Get that wrong and you remove yourself from AI answers while leaving model training untouched — which is the exact opposite of what most people intend.
What llms.txt is
A plain-text or Markdown file at your site root — /llms.txt — listing your most important pages with short descriptions, so a language model can find the useful parts of your site without crawling all of it. Some sites also publish /llms-full.txt containing the content itself.
The idea is reasonable. It is essentially a sitemap written for a reader that thinks in prose rather than URLs.
The problem is adoption on the consuming side. A file only works if something reads it, and the largest system that would need to read it has said clearly that it does not.
Where the evidence actually stands
Worth separating carefully, because this is the part every other article gets muddled.
| Platform | Position |
|---|---|
| Does not use it. Stated repeatedly, and named as unnecessary in its 15 May 2026 AI features guidance | |
| Anthropic | Recommends llms.txt in its guidance on writing documentation for agents |
| OpenAI | Publishes llms.txt files for its own developer products |
| Perplexity | Reported to surface llms.txt content, though not formally documented |
Read that table twice, because the pattern in it is the actual answer. The places llms.txt is genuinely used are agent and developer workflows — an AI agent working with your documentation, a coding assistant reading your API reference. That is a real and growing use case.
None of it is search visibility. Publishing llms.txt will not get you into an AI Overview, will not affect your ranking, and will not make ChatGPT more likely to cite your blog post.
Two further cautions. Claims that a given assistant "respects llms.txt directives in retrieval" circulate widely and are mostly inference from correlation rather than documented behaviour — treat them as unverified. And Google indexing tens of thousands of these files is not evidence that anything uses them; Google indexes text files.
So should you publish one?
If you have developer documentation, an API, or a product agents interact with: yes. It is cheap and the use case is real.
If you run a marketing site or a blog and you want AI visibility: it will not do that. Publish one if you like — it costs nothing and harms nothing — but do not let anyone bill you for it as an AI SEO deliverable, and do not expect it to show up in any metric.
Keep it accurate if you publish it. A stale llms.txt pointing at pages that have moved is worse than none, for exactly the same reason a stale sitemap is.
The crawler decision that actually matters
Three kinds of bot reach your site, they do different jobs, and robots.txt treats them identically unless you separate them yourself.
Training crawlers
They collect content to train future models. Blocking them affects what future models know about you. It has no effect on anything happening today.
| User-agent | Operator |
|---|---|
GPTBot |
OpenAI |
ClaudeBot |
Anthropic |
Meta-ExternalAgent |
Meta |
CCBot |
Common Crawl |
Bytespider |
ByteDance |
Google-Extended |
Google — an opt-out token, not a crawler |
Applebot-Extended |
Apple — an opt-out token, not a crawler |
Google-Extended and Applebot-Extended are not crawlers. They are tokens you put in robots.txt to opt out of AI training, and they do not affect Googlebot or Applebot fetching your pages for search. Disallowing Google-Extended does not remove you from Google Search or from AI Overviews.
Retrieval and search crawlers
They build the indexes assistants search when answering a question. These are the ones that decide whether you can be cited today.
| User-agent | Operator |
|---|---|
OAI-SearchBot |
OpenAI |
Claude-SearchBot |
Anthropic |
PerplexityBot |
Perplexity |
Googlebot |
Google — search, AI Overviews and AI Mode |
User-triggered fetchers
They fetch a page because a person asked the assistant about it — pasted a link, or asked a question the assistant needed a live page to answer.
| User-agent | Operator |
|---|---|
ChatGPT-User |
OpenAI |
Claude-User |
Anthropic |
Perplexity-User |
Perplexity |
Blocking these means that when someone asks an assistant about your page, the assistant cannot read it. That is rarely what anyone wants, and it is the most commonly misunderstood category of the three.
The mistake almost everyone makes
Blanket-blocking everything with "AI" or "GPT" in the name.
The intention is usually "don't train on my work." The effect is usually "remove me from AI answers," because the same copy-pasted robots.txt block catches OAI-SearchBot and ChatGPT-User alongside GPTBot.
The result is a site that is still in the training data collected before the block, and no longer eligible to be cited in the answers people are getting now. You gave up the visibility and kept the exposure.
The reverse mistake is quieter: disallowing Google-Extended in the belief it opts you out of AI Overviews. It does not. AI Overviews are generated from Google's search index, fetched by Googlebot, and robots.txt has never offered a way to opt out of them.
Google now offers that opt-out directly
This is new enough that most guidance predates it. Alongside its AI performance reporting, Google rolled out a Search generative AI control globally on 31 August 2026 — a first-party setting for whether your site appears in and helps ground its generative AI Search features.
Three things about it matter.
- It covers AI Overviews, AI Mode, and generative AI in Discover — the surfaces robots.txt could never reach.
- Opting out forfeits the impressions along with the traffic. You do not become invisible while staying eligible; you leave the surface entirely.
- Google states it is not used as a ranking signal for search results outside those generative AI features.
This is the control most people were reaching for when they blocked Google-Extended. If your objection is to being summarised in Google's AI answers, this is the setting that expresses it — and the honest trade is that you lose the visibility too, at a point where AI impressions are the fastest-growing thing Search Console reports.
Decide it deliberately rather than by default. The reason to opt out is a considered position about your content being summarised. The reason not to is that being absent from the answer does not send the reader to your page instead — it sends them to whoever stayed.
How to decide
Four questions, in order.
1. Does your business depend on people finding you?
If yes, allow every retrieval and user-triggered crawler. Being absent from AI answers is a distribution decision, not a rights decision.
2. Is your content the product?
A paywalled archive, a subscription publication, a licensed dataset — training crawlers are a genuine commercial question and blocking them is defensible. Note that it protects future training only; anything already collected is collected.
3. Is your objection to training specifically?
Then block training crawlers and allow retrieval crawlers. This is the position most publishers actually want and the one the copy-paste blocks fail to express.
4. Do you need enforcement rather than a request?
robots.txt is a convention, not a control. Compliant operators honour it; others do not, and Bytespider has a documented reputation for ignoring it. If you need enforcement, that is server-level blocking or a CDN rule, not a text file.
The robots.txt that expresses "cite me, don't train on me"
The most common intent, written correctly:
# Training — disallowed
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Retrieval and citation — allowed
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /
Three things to check before you deploy it. Confirm you have not also disallowed Googlebot anywhere in the file. Verify the file parses — a syntax error higher up can invalidate the rules below it. And re-read your user-agent strings against the operators' current documentation, because this list changes and a misspelled agent name is silently ignored.
A blanket User-agent: * rule does not do what you want here. It catches Googlebot too.
Verify what is actually reaching you
Your server logs settle every question this page raises, and they are free.
Grep your access logs for the user-agent strings above. You will typically find far fewer AI crawler hits than expected, and the distribution is usually surprising — retrieval bots active, training bots occasional. That distribution is the only data on this subject that is genuinely about your site, and it beats any third-party estimate.
Bot hits are invisible to Google Analytics, because bots do not run JavaScript. If you have never looked, that is the gap between what you think is happening and what is.
Frequently asked questions
Does Google use llms.txt?
No. Google has said so repeatedly, John Mueller has compared it to the keywords meta tag, and Google's May 2026 AI features guidance names the file as unnecessary for AI Overviews and AI Mode. Publishing one has no effect on Google Search visibility.
Is llms.txt worth creating?
For developer documentation, APIs, or products that AI agents interact with, yes — Anthropic recommends it for agent-facing documentation and OpenAI publishes them for its own developer products. For a marketing site or blog hoping for AI search visibility, no. It costs nothing and does nothing.
Should I block AI crawlers in robots.txt?
Only after separating them by purpose. Training crawlers feed future models; retrieval crawlers decide whether an assistant can cite you today; user-triggered fetchers read a page when someone asks about it. Blocking all three at once removes you from AI answers while leaving already-collected training data untouched.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended is an opt-out token for AI model training, not a crawler, and AI Overviews are generated from the search index Googlebot builds. Since 31 August 2026 Google offers a separate Search generative AI control for opting out of AI Overviews, AI Mode and generative AI in Discover. Opting out forfeits the impressions as well as the traffic, and Google states it is not used as a ranking signal elsewhere.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content for model training. OAI-SearchBot builds the search index ChatGPT uses when answering questions with live sources. Blocking GPTBot affects future models; blocking OAI-SearchBot removes you from citations in ChatGPT answers now.
Do AI crawlers respect robots.txt?
The major operators state that they do, and their crawlers generally behave accordingly. Compliance is voluntary — Bytespider in particular has a documented reputation for ignoring it. If you need enforcement rather than a request, that is server-level or CDN blocking.
How do I see which AI crawlers visit my site?
Search your server access logs for the user-agent strings. Analytics will not show them, because bots do not execute JavaScript. Your logs are the only source of data about crawler behaviour on your own site, and they usually contradict expectations.
Will blocking AI crawlers protect my content from being used?
Only partially and only going forward. It stops compliant crawlers collecting new content; it does nothing about content already collected, content republished elsewhere, or operators that ignore robots.txt. It is a request, not a protection.
What to do next
Open your robots.txt and check whether you have blanket-blocked anything with "GPT" or "AI" in the name. If you have, you have probably removed yourself from AI citations without stopping training in any meaningful way.
Then grep your access logs for OAI-SearchBot, PerplexityBot, ClaudeBot and GPTBot. Ten minutes, and it replaces every estimate on this subject with your own data.
Related guides
- SEO when AI answers the question first — what actually affects AI visibility
- Entity SEO: why named things beat keywords now — what does work
- How to track AI search traffic in GA4 — measuring what gets through
- Content formats that still get clicks from AI — what gets cited once you are crawlable
- robots.txt mistakes that de-index whole sites — the syntax errors that cost most
Free: The 60-Minute Email Authentication Fix
A no-fluff checklist to set up SPF, DKIM & DMARC correctly and pass Gmail & Yahoo's sender requirements.
Muhammad Basim has worked in digital marketing since 2013, focused on email deliverability and AI-assisted content production. He is the author of The Email Deliverability Playbook and The Email Copywriting Playbook.
Related Articles
Email Deliverability: Why Authenticated Emails Still Land in Spam
Email deliverability is whether your message reaches the inbox rather than the spam folder, and it is decided by three factors: technical infrastructure, list quality, and sending behaviour. Authentication is one item inside the first factor. Getting it right is necessary, and it is nowhere near sufficient. That gap explains the most common complaint in […]
Linkable Assets: What Actually Earns Links
A link is a citation, and a citation requires that someone writing about your subject needed you to make their point. That is the whole mechanism, and it is the reason most "linkable content" earns nothing. The test, before you build anything: Could a writer covering this topic finish their sentence without referencing you? If […]
Link Outreach Emails That Get Replies
Outreach fails for exactly two reasons, and only one of them gets written about. The first is that the email was not worth replying to. That is the reason every guide addresses, and the advice — personalise, be brief, offer value — is correct and insufficient. The second is that the email never arrived. Nobody […]