If AI assistants cannot read your website, they cannot recommend your business. That single sentence is why your robots.txt file — a plain text file most owners have never opened — quietly decides part of your AI visibility. Many sites block AI crawlers without knowing it: a CDN default, a plugin's “block AI bots” toggle, or a robots.txt rule copied from a blog post in 2023 can make you invisible to the systems your customers now ask for recommendations.

This is a reference list of the AI crawlers that matter in 2026: who operates each one, what it actually does with your content, and which ones a business that wants AI visibility should allow. Every user agent below is documented by its operator; where an operator publishes no documentation, we say so.

First, Understand the Three Kinds of AI Bot

“AI crawler” covers three very different behaviors, and the right policy differs by kind:

  • Training crawlers collect content to train future models. Blocking them limits what tomorrow's models know about you; the effect is slow and diffuse.
  • Search-index crawlers build the retrieval indexes that AI search products cite live. Blocking them removes you from AI answers that ground on the web — a fast, direct hit to visibility.
  • User-request fetchers visit a page because a human asked the assistant about it right now. Blocking them means the assistant cannot verify what you say — including when a potential customer asks “is this business legit?”

For a business that wants customers, the last two categories are close to non-negotiable. The first is a genuine policy choice.

OpenAI (ChatGPT)

OpenAI documents three bots (official documentation):

  • GPTBot — the training crawler. Allowing it lets your content inform future GPT models.
  • OAI-SearchBot — builds the search index behind ChatGPT's web search. Blocking it can keep you out of ChatGPT answers that link to sources. If AI visibility is the goal, allow it.
  • ChatGPT-User — fetches pages when a user's request requires visiting your site. Allow it: this is often a real prospect, one question away from contacting you.

Anthropic (Claude)

Anthropic likewise documents its crawlers and confirms they respect robots.txt (official documentation):

  • ClaudeBot — the training crawler.
  • Claude-User — fetches pages in response to a user's live request.
  • Claude-SearchBot — indexes content to improve Claude's search results.

The same logic applies as with OpenAI: the user and search bots are the ones directly tied to whether Claude can read and cite you today.

Perplexity

Perplexity is an AI search engine, so its crawlers are all about live answers (official documentation):

  • PerplexityBot — builds Perplexity's search index.
  • Perplexity-User — visits a page when a user's question requires it.

Blocking these removes you from one of the most citation-heavy AI surfaces there is — Perplexity answers link their sources prominently, which is exactly the kind of exposure a business wants.

Google (Gemini and AI Features)

Google-Extended is not a separate crawler — it is a robots.txt token that controls whether content Googlebot already crawled may be used for training Gemini models (official documentation). Two details matter:

  • Blocking Google-Extended does not affect your Google Search rankings or whether you appear in AI Overviews — those are governed by normal Googlebot rules.
  • Allowing it costs nothing extra in crawl traffic, since it piggybacks on Googlebot's existing crawl.

The Rest of the List

  • CCBot (Common Crawl) — a nonprofit crawler building an open web archive (official documentation). Not an AI company itself, but Common Crawl data has historically been a major training source for many models — blocking CCBot reduces your presence in the datasets many labs start from.
  • Meta-ExternalAgent (Meta) — Meta's crawler for AI training and product improvement (official documentation), relevant to Meta AI inside WhatsApp, Instagram, and Facebook.
  • Bytespider (ByteDance) — ByteDance's crawler, associated with training its models. ByteDance publishes no official documentation page for it, and it has a reputation for aggressive crawling; whether it honors robots.txt is not something the operator publicly confirms. We list it for completeness rather than endorsement.

Other agents you may see in logs — Applebot-Extended (Apple's AI-training opt-out token), Amazonbot, DuckAssistBot and various startup crawlers — follow the same three-way logic: check the operator's documentation, identify whether it is training, search, or user-request, and decide accordingly.

A Sensible Default Policy

For a business whose goal is to be found and recommended, allow all search-index and user-request bots, and allow the training bots of the major assistants your customers use:

  • Allow: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended.
  • Judgment call: CCBot and Meta-ExternalAgent (training-oriented; allowing them is a reasonable default for visibility, blocking them is a defensible content-rights choice).
  • Judgment call: Bytespider (undocumented; block it if its traffic bothers you — for most Western small businesses it has little visibility upside).

Since robots.txt rules are allow-by-default, the most common fix is not adding Allow lines — it is removing an overly broad Disallow: / that a security plugin or CDN preset added for these user agents. If you do want explicit rules, this is the minimal pattern:

User-agent: GPTBot
Allow: /

…repeated per bot. One warning from sites we have scanned: some firewalls block AI crawlers at the network level even when robots.txt allows them, so a clean robots.txt does not guarantee access. Test the real behavior, not just the file.

Check Your Site in 30 Seconds

You can audit your robots.txt against this whole list with our free AI crawler checker — paste your domain and it shows, bot by bot, who is allowed, who is blocked, and by which rule.

If you prefer the terminal (or want this in CI so a deploy never silently blocks AI again), the same check ships in our open-source CLI (GitHub): npx rankedbyai crawlers yoursite.com exits non-zero when any of the 11 major AI crawlers is blocked. And to see what AI assistants actually say about your business once they can read you, run a free AI visibility scan.

FAQ

Should I ever block AI crawlers?

It depends on what your content is worth to you. Publishers whose product is the content itself often block training crawlers to protect licensing value. A local business is the opposite case: your content exists to be found, so blocking the systems customers ask for recommendations mostly hurts you. If you want a middle path, allow search and user-request bots but block training bots — the categories above make that split easy.

Does allowing AI crawlers hurt my Google rankings?

No. AI crawler rules in robots.txt are per-user-agent and do not affect Googlebot. Google-Extended in particular is explicitly documented as having no effect on Search ranking — it only controls Gemini model training.

How do I know if AI crawlers are being blocked right now?

Check your robots.txt for Disallow rules targeting the user agents above (including a blanket User-agent: * / Disallow: /), then verify with the crawler checker. Also check your CDN or security settings — Cloudflare, for example, offers a one-click AI-bot block that operates independently of robots.txt.

Do all these bots actually respect robots.txt?

OpenAI, Anthropic, Perplexity, Google, Common Crawl, and Meta all document that their listed crawlers honor robots.txt. Bytespider's compliance is not officially documented, and independent site owners have reported inconsistent behavior — if blocking it matters to you, consider a firewall rule rather than trusting robots.txt alone.