Should You Block AI Crawlers? robots.txt for Business Sites
Each AI company runs separate bots for training, search and user requests, and blocking the wrong one hides you from AI answers without stopping training. The bots, three ready robots.txt files, and how to check what yours does.
For most small business sites, the answer is: allow the search bots and the user-triggered fetchers, and decide separately about the training bots. Each AI company runs more than one crawler, each with a different job. A blanket "block AI" rule in robots.txt removes you from AI answers and does little about training, since models already trained on your pages keep what they learned.
This post walks through the bots, what each one does, how to write the rules for three common goals, and how to check whether your rules are actually working. It is part of our guide to agent-ready websites.
Why this is not one decision
"AI crawlers" sounds like one thing. It is at least three:
- Training crawlers collect pages that may be used to train future models.
- Search crawlers build the index an assistant searches when it answers, and decide whether you can be cited.
- User-triggered fetchers load a specific page because a person asked the assistant about it right now.
Blocking the first does not affect the second or third. Blocking the second takes you out of the assistant's answers. Blocking the third stops the assistant from reading your page even when a customer pastes your link into the chat. A business that wants to be found through AI assistants wants the second and third, whatever it decides about the first.
It also helps to know what blocking cannot do. A training block only applies from the day the crawler next reads your robots.txt. It does not remove your pages from models that have already been trained, and it does not reach copies of your site that already sit in public archives.
The bots, by company
| Company | Training | Search | User-triggered |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | none documented | PerplexityBot | Perplexity-User |
| Google-Extended (token) | Googlebot | Google-Agent and other user-triggered fetchers | |
| Microsoft | none documented | Bingbot, which also grounds Copilot | none documented |
| Apple | Applebot-Extended (token) | Applebot | none documented |
| Meta | meta-externalagent | none documented | meta-externalfetcher |
| Common Crawl | CCBot | none | none |
| ByteDance | Bytespider | none | none |
OpenAI documents its three agents in its crawler overview. Each setting is independent, so a site can allow OAI-SearchBot for ChatGPT search while disallowing GPTBot. OpenAI notes that ChatGPT-User acts on a user's request, so robots.txt rules may not apply to it in the same way, and it publishes the IP ranges each bot uses.
Anthropic describes ClaudeBot, Claude-SearchBot and Claude-User in its support article, and notes that disabling Claude-SearchBot or Claude-User may reduce your visibility in Claude's answers.
Perplexity lists PerplexityBot, which surfaces and links sites in its results, and Perplexity-User for user requests in its crawler documentation. It states that Perplexity-User generally ignores robots.txt because a person asked for the page.
Google uses Googlebot for Search, including AI Overviews and AI Mode. Google-Extended is not a separate crawler but a token in robots.txt that controls whether content is used for Gemini apps and model training. Google's crawler documentation states it does not affect inclusion or ranking in Search, so blocking Google-Extended does not keep a page out of AI Overviews. In robots.txt, the only way to do that is to block Googlebot, which removes you from Search entirely. Since August 2026 there is a better option outside robots.txt: the Search generative AI control in Search Console excludes a whole site from AI Overviews, AI Mode and generative AI features in Discover, and Google says it is not used as a ranking signal for the rest of Search. For single pages or passages, Google's AI features guidance points to the nosnippet, data-nosnippet and max-snippet controls. Google lists newer agents, such as Google-Agent, in its crawling changelog.
Microsoft does not document a separate AI crawler. Copilot and Bing's AI summaries draw on the Bing index that Bingbot builds, so blocking Bingbot takes you out of Bing search and Copilot answers together.
Apple uses Applebot for Siri and Spotlight, and an Applebot-Extended token for training, which works the same way as Google-Extended.
Common Crawl runs CCBot, which builds a free public archive of the web. Many model builders train on that archive, so blocking CCBot is the closest thing to a single opt-out from open training data.
Three goals, three robots.txt files
Below are three starting points. Each is a complete robots.txt you can adapt. Replace the sitemap address with your own.
Goal 1: be found everywhere, including training. This is the simplest case and the default on many platforms.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlGoal 2: be found and cited by AI assistants, but stay out of training. This is the choice most business sites with original writing end up making.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /
Sitemap: https://example.com/sitemap.xmlGoal 3: classic search only, no AI crawlers at all. Googlebot, Bingbot and Applebot keep working. Your site will not be cited in ChatGPT, Claude or Perplexity answers. Google's AI Overviews and AI Mode can still draw on it, unless you also exclude the site with the Search generative AI control in Search Console. Copilot can too, because it uses the Bing index.
User-agent: *
Content-Signal: search=yes, ai-input=no, ai-train=no
Allow: /
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /
Sitemap: https://example.com/sitemap.xmlGrouping several User-agent lines above one rule is valid under the robots.txt standard, RFC 9309. A crawler follows the most specific group that names it, so these rules sit safely alongside the general User-agent: * block.
The Content-Signal line
In 2025 Cloudflare introduced the Content Signals Policy, an addition to robots.txt with three signals. search covers building a search index and showing links. ai-input covers feeding content into an AI answer at the moment someone asks. ai-train covers training or fine-tuning models.
Its value is that it states your intent in one readable line, which per-bot rules cannot do for bots you have never heard of. Its limit is that it depends on companies choosing to honor it, and none of the large AI companies has publicly committed to that yet. Treat it as a clear statement of your terms, not as enforcement.
A standards-track version of the same idea is under way. The IETF's AI Preferences working group is drafting a shared vocabulary with terms for AI training, AI use and search, plus a way to attach it to content through robots.txt and HTTP headers. It is still an Internet-Draft, not a finished standard. If it is adopted, expect the Content-Signal line to give way to it.
robots.txt is a request, not a lock
Reputable crawlers follow robots.txt, and the major AI companies say theirs do. Not every crawler is reputable, and user-agent names can be copied by anyone.
In August 2025 Cloudflare published evidence that Perplexity fetched pages on test domains that disallowed it, using a generic Chrome user-agent and rotating IP addresses when its declared crawler was blocked. Perplexity disputed the findings. Either way, the episode showed the limit of a text file.
If you need real enforcement, it happens in front of the site:
- Verify bots by IP, not by name. OpenAI, Google and others publish the IP ranges their crawlers use. A request that claims to be GPTBot from an address outside those ranges is not GPTBot.
- Check signatures where they exist. A newer method, Web Bot Auth, has agents sign each request with HTTP Message Signatures (RFC 9421), which a site or CDN can verify against the operator's published keys. OpenAI signs requests from ChatGPT's cloud browser this way. A signature is much harder to fake than a name or an IP address.
- Use a firewall or CDN rule. Cloudflare and similar services can block or challenge AI crawlers at the network level. Since July 1, 2025, Cloudflare asks new domains during setup how to treat AI crawlers, and its defaults block training crawlers while leaving search crawlers allowed. A site behind it may be blocking bots you meant to allow, so check the setting.
What you give up, and what you get
The trade is easier to judge with numbers. Cloudflare measures how many pages each AI platform fetches for every visitor it sends back. In its analysis of the first week of August 2025, that ratio was close to 50,000 to 1 for Anthropic, 887 to 1 for OpenAI and 118 to 1 for Perplexity. Most of that crawling is for training, which sends no visitors at all.
So for a business site the picture is:
- Search and user-triggered bots cost little bandwidth and are the only route to being cited and clicked in AI answers. Allowing them is almost always worth it.
- Training bots send nothing back directly. Allowing them may help a model know your business exists when it answers without searching. Blocking them protects original writing from being absorbed. Both are defensible.
- A blanket block gives up the first to stop the second, and usually fails at the second anyway, because your pages are likely already in older training data.
Publishers with large original archives face a different calculation, with a stronger case for blocking training and for licensing deals. This post is about business sites.
How to check what you have now
- Read your robots.txt. Open yourdomain.com/robots.txt. Look for
Disallow: /underUser-agent: *, which blocks everything including search engines, and check whether search bots are blocked alongside training bots. - Look for rules you did not write. SEO plugins and CDNs often add their own. Cloudflare's managed robots.txt, for example, prepends rules when enabled.
- Test it. Google Search Console has a robots.txt report showing what Google fetched and any errors. While you are there, check the Search generative AI setting, so you know whether the site is included in AI Overviews and AI Mode.
- Check your logs. Search your server logs or analytics for the bot names above. If OAI-SearchBot or PerplexityBot never appears, something other than robots.txt may be blocking it, such as a firewall rule.
- Ask the assistants. The quickest end-to-end test is to ask about your business and see whether your site is cited. The method is in How to Audit What ChatGPT Says About Your Business.
How this works in Beste
Every Beste site publishes a robots.txt with a Content-Signal line that allows AI search and AI input by default, so the starting point is to be found. If you want Goal 2 or Goal 3 from above, edit robots.txt in SEO settings and paste your own rules, including per-bot blocks. The technical SEO page describes the editor, and the agent-ready websites page lists what else is published alongside.
Whatever you decide about crawling, the next question is whether the bots you let in can tell who you are. That is the subject of Entity Schema for Small Sites. If you are wondering whether you also need an llms.txt file, see llms.txt Explained: What the Data Says and Who Needs One.
Sources
Checked in October 2026. Bot names and policies change, so check each company's page before you edit robots.txt.