How to allow AI crawlers (and which ones to block)

What an AI crawler is and why robots.txt matters

An AI crawler is an automated program that visits web pages on behalf of an AI company. Some collect text to train models. Some build a search index that an assistant draws on. Some fetch a single page because a person asked a question.

The file that tells them what they may visit is robots.txt. It is a plain text file at the top of your site, for example yourdomain.com/robots.txt. The standard behind it, RFC 9309, describes it as a set of rules that well-behaved crawlers follow. The same document states plainly that these rules are not a form of access authorization.

That is the first thing to understand. Robots.txt asks politely. It does not lock anything. Reputable companies honor it, and the rest can ignore it. If a page is private, protect it with a login, not with a robots.txt line.

It matters to you because an assistant cannot cite a page it was never allowed to read. If your file or your firewall turns the wrong bot away, you can disappear from answers without anyone noticing.

The three jobs an AI bot can do

Most confusion comes from treating all AI bots as one thing. The big companies now separate them, and the difference decides what you should allow. Think of three jobs.

• Training. A training crawler gathers content that may be used to train future models. Allowing it is a decision about your content being used to build the model, not about showing up in answers today.
• Search. A search crawler builds the index an assistant searches when it answers a question. Allowing it is what makes your pages eligible to be cited.
• User-triggered fetching. A user-triggered fetcher visits a page because someone asked the assistant about it or about a topic it covers. It is not a crawler that roams your site.

The bots to know, from each company’s own documentation

These descriptions follow the vendors’ pages as of their publication. Names and behavior change, so check the source before you edit your file.

• OpenAI documents three bots. GPTBot is for content that may train its models. OAI-SearchBot is for ChatGPT search, and OpenAI says sites that opt out will not appear in ChatGPT search answers. ChatGPT-User handles actions started by a user, and OpenAI says robots.txt rules may not apply to it.
• Anthropic documents ClaudeBot for training, Claude-User for fetching pages when a Claude user asks a question, and Claude-SearchBot for improving the relevance and accuracy of search results. Anthropic says its bots follow robots.txt and support the Crawl-delay extension.
• Perplexity documents PerplexityBot, which it says surfaces and links sites in its search results and is not used to crawl content for foundation models. Perplexity-User is a user-triggered fetcher that supports user actions and generally ignores robots.txt because a person requested the fetch.
• Google-Extended is a robots.txt control for Google, not a separate crawler. Google says it governs use of your content for training future Gemini models and for grounding in Gemini Apps, and that it does not affect inclusion in Google Search.

How to allow the crawlers you want, step by step

Open yourdomain.com/robots.txt in a browser first. Many sites already have a file written by a plugin or a developer years ago, and it may contain a blanket rule that catches AI bots.

A group starts with a User-agent line naming the bot, followed by rules. If a bot has no group of its own, it falls back to the group for the wildcard * if one exists. When several groups apply, the most specific path rule wins. That is why an old Disallow: / under User-agent: * can block every bot you did not name.

• To allow ChatGPT search: write User-agent: OAI-SearchBot followed by Allow: /.
• To allow Claude search and user fetches: write one group each for Claude-SearchBot and Claude-User with Allow: /.
• To allow Perplexity: write User-agent: PerplexityBot followed by Allow: /.
• To opt out of training only: write User-agent: GPTBot, then Disallow: /, and do the same for ClaudeBot and Google-Extended.
• Repeat the file on every subdomain you want covered, because each host has its own robots.txt.

The step most sites miss: your firewall and CDN

A correct robots.txt does nothing if your security layer refuses the bot before it reads the file. Content delivery networks and web application firewalls often ship with managed rules that challenge or block AI user agents by default.

Anthropic notes that blocking by IP address may not work reliably, because it stops the bot from reading your robots.txt at all. Perplexity publishes machine-readable lists of its IP ranges and says to treat them as the source of truth because they update regularly. Anthropic publishes a list too.

The practical step is to check your firewall or CDN dashboard for bot rules, allow each search crawler you chose, and verify a bot by its published ranges rather than by the name it claims. Anyone can pretend to be a bot in a request header.

Deciding about training bots

Blocking a training crawler is a legitimate choice, and many publishers make it. Because the vendors separate training from search, you can usually make that choice without leaving answers. OpenAI says disallowing GPTBot signals that your content should not be used for training, and OpenAI says its search bot is for search.

Weigh two things. If your content is your product, such as paid research, you have a clear reason to opt out. If your content exists to win customers, being learned by a model and mentioned by it may help you, but no one can promise that. Decide on purpose, write it down, and revisit it when the vendors change their policies.

Also remember Google’s position: the Google AI features page says there are no extra requirements to appear in AI Overviews or AI Mode beyond being indexed and eligible to show a snippet. Blocking Google-Extended is not how you leave those features. Blocking Googlebot is, and it removes you from Search too.

How to check that it worked

Start with the file itself. Load it, confirm it returns a normal page, and read it top to bottom. Under RFC 9309 a 4xx response means the file is treated as missing, so crawlers may visit anything. A 5xx response or timeout means crawlers must assume everything is disallowed, which is the failure you least want.

Next read your server logs or ask your host for them. Search for the bot names above and look at the status codes. A 200 means the bot got the page. A 403 or 429 means something is blocking it, often the firewall. No hits at all after a week can mean the bot has not found you yet, which is normal for a small site.

Then wait. OpenAI says its systems may take about 24 hours to adjust after a robots.txt change, and Perplexity gives the same figure. Re-test the following week, and re-test after any site migration or security change.