AI Crawlers in robots.txt: Which to Allow and Which to Block in 2026

For twenty-five years, robots.txt answered one question: should search engines index this page? In 2026 it answers a harder one. A dozen AI companies now send crawlers to your site, and they don't all want the same thing. Some feed live answers in ChatGPT and Perplexity — and link back to you. Others quietly vacuum your writing into a training set you'll never see again. Blocking all of them makes you invisible to AI search; allowing all of them hands your work to model trainers for free.

The good news: you can tell them apart, and robots.txt is still where you draw the line. This guide covers which AI crawlers exist in 2026, what each one actually does, and exactly what to put in your file — whether your goal is maximum AI-search visibility, strict content protection, or a middle path.

The three jobs AI crawlers do

"AI bot" is a useless label, because it hides three very different behaviours. Getting your policy right depends entirely on separating them.

Type	What it does	Sends you traffic?
Search / retrieval	Indexes your page so an AI answer can cite it — with a clickable link back to you.	Yes — this is the AI-search referral channel.
User-triggered fetch	Grabs a specific URL a real person pasted or asked the assistant to open, right now.	Sometimes — a human is on the other end.
Training	Collects text to train or fine-tune a model, permanently absorbing it. No link, no attribution.	No — this is one-way extraction.

The strategic insight that runs through this whole guide: block training, keep search. Blocking a training crawler costs you almost nothing in visibility, because that bot was never going to send a reader your way. Blocking a search crawler removes you from the AI answers that do link out. Confuse the two and you either give your content away or delete yourself from the fastest-growing referral source of the year.

The 2026 AI crawler cheat sheet

Here are the user-agents that matter today, grouped by who operates them. Match on these strings in robots.txt. Note that most vendors run separate bots for separate jobs — that separation is what makes selective policies possible.

Operator	User-agent	Job
OpenAI	GPTBot	Training
OpenAI	OAI-SearchBot	Search (ChatGPT Search)
OpenAI	ChatGPT-User	User-triggered fetch
Google	Google-Extended	Training (Gemini / Vertex)
Google	Googlebot	Search (also feeds AI Overviews)
Anthropic	ClaudeBot	Training
Anthropic	Claude-SearchBot	Search
Anthropic	Claude-User	User-triggered fetch
Perplexity	PerplexityBot	Search / index
Perplexity	Perplexity-User	User-triggered fetch
Apple	Applebot-Extended	Training opt-out token
Meta	meta-externalagent	Training
Common Crawl	CCBot	Training data (feeds many models)
ByteDance	Bytespider	Training (often ignores rules)

Two of these deserve a flag. Google-Extended is not a real crawler — it's a permission token. Disallowing it opts you out of Gemini training without touching Googlebot, so your normal Google ranking and AI Overviews eligibility are untouched. Applebot-Extended works the same way for Apple Intelligence. And Bytespider is the one bad actor worth singling out: it has been reported to account for a huge share of AI crawl traffic and to frequently ignore disallow rules. Block it, but don't be surprised if it knocks anyway.

Pick your policy

There is no single "correct" robots.txt — only the one that matches your goals. Three stances cover almost everyone.

Policy A — Maximum AI visibility (recommended for most publishers)

You want to be cited everywhere an AI answer links out, and you're relaxed about training. Do nothing restrictive — just make sure you aren't accidentally blocking search bots. This is the right default if traffic and brand reach matter more to you than controlling model training.

# Everyone welcome (default). Nothing to block.
User-agent: *
Allow: /

Policy B — Stay cited, refuse training (the balanced choice)

This is the sweet spot for most serious content brands: keep the search and retrieval bots that drive AI-search citations, and shut the door on pure training crawlers. You lose almost no visibility and stop feeding models for free.

# Block training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# Explicitly welcome search & retrieval
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Perplexity-User
Allow: /

Policy C — Lock it down

You publish paywalled, proprietary, or licensed content and want no AI system reading it without a deal. Block search and training alike — and accept that you will not appear in AI answers.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /

One caution for Policy C: do not disallow Googlebot here unless you also want to vanish from ordinary Google Search. Keep classic search separate from AI opt-outs.

Five mistakes that quietly break your setup

1. Blocking Googlebot to "block AI." Googlebot is your normal search crawler. Disallow it and you delete your organic rankings, not just AI Overviews. Use Google-Extended for the AI-training opt-out.
2. Treating robots.txt as a lock. It's a request, not a firewall. Reputable bots comply; bad actors like Bytespider may not. If you truly must stop a bot, enforce it at the network or CDN level, as Cloudflare's default AI-blocking now does.
3. Blocking retrieval bots by accident. A broad User-agent: * disallow, or an over-eager "block all AI" rule, can catch OAI-SearchBot and PerplexityBot — the exact crawlers that get you cited. Always list search bots explicitly if you block anything.
4. Confusing training tokens with crawlers. Google-Extended and Applebot-Extended don't fetch anything; they're consent flags. Disallowing them is safe and precise — it never affects search indexing.
5. Forgetting robots.txt is public. Anyone can read yours at /robots.txt. Never use it to hint at private URLs — you're publishing a map of what you'd rather hide.

robots.txt is not your whole AI strategy

Controlling crawler access decides whether AI systems can read you. It does nothing to make them cite you once they can. Those are separate problems. If your goal is to show up in AI answers, opening the gate is step one; being quotable is step two — and that is won with clean, server-rendered HTML, clear passage-level answers, and strong optimization across each AI engine, since ChatGPT, Perplexity and Google draw from different sources.

It's also worth knowing what robots.txt can't express. It's a coarse allow/deny list with no way to say "index this but don't train on it" beyond the vendor-specific tokens above. That gap is why newer signals like llms.txt and network-level permission layers are emerging — attempts to add nuance the original protocol never had. For now, though, robots.txt remains the first and most widely respected control you have. Set it deliberately, review it whenever a new major crawler appears, and treat "who gets to read my site, and for what" as a decision you make — not a default you inherited.

FAQ

Will blocking GPTBot remove me from ChatGPT search results?

No. GPTBot is OpenAI's training crawler. ChatGPT's live search results are gathered by OAI-SearchBot, a separate bot. You can disallow GPTBot to opt out of training while allowing OAI-SearchBot to stay eligible for citations with links back to your site.

Does robots.txt actually stop AI companies from scraping my content?

Only the ones that choose to comply. Major operators like OpenAI, Anthropic, Google and Perplexity respect robots.txt. Some crawlers — Bytespider is the frequently cited example — have been reported to ignore disallow rules, so if you need hard enforcement, block at the CDN or firewall level rather than relying on robots.txt alone.

What is the difference between Google-Extended and Googlebot?

Googlebot is Google's search crawler that powers ordinary rankings and AI Overviews. Google-Extended is a separate permission token that controls whether your content trains Gemini and Vertex AI models. Disallowing Google-Extended opts you out of AI training without affecting your normal Google Search visibility.

Should I block AI training crawlers?

It depends on your goals. Blocking training bots such as GPTBot, ClaudeBot and Google-Extended costs you very little visibility, because those crawlers don't send referral traffic. If your content is distinctive or licensed, blocking training while keeping search bots open is a sensible balance. If broad AI exposure helps you, you can leave everything open.

How often should I update my robots.txt for AI bots?

Review it whenever a major AI vendor launches or renames a crawler, and at least once a quarter. New user-agents appear regularly, and vendors occasionally split a single bot into separate search and training crawlers — which changes what a given rule actually does.