You can allow AI search crawling while restricting training collection, but the controls depend on the provider. OpenAI explicitly separates OAI-SearchBot from GPTBot. Allowing the former does not require allowing the latter, according to OpenAI's crawler documentation.
The important exception is Google: its Google-Extended control covers specified Gemini grounding uses as well as training. It is not a training-only switch, according to Google's crawler reference.
For a business website, start with the uses you want to permit: search discovery, training collection, grounding and user-requested page visits. Then map that policy to each provider's controls.
Separate search bots from training bots
OpenAI and Anthropic document distinct bots for search and potential model-training collection:
| Provider | Search bot | Training bot | What blocking search changes |
|---|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot | Excludes the site from ChatGPT search answers, although navigational links may remain. |
| Anthropic | Claude-SearchBot | ClaudeBot | Prevents indexing for search optimization and may reduce search visibility. |
These roles and consequences come from OpenAI's bot definitions and Anthropic's crawler guidance.
Anthropic says restricting ClaudeBot signals that future materials should be excluded from its training datasets. That is a prospective opt-out, not a promise to remove content from existing models. Anthropic also says its bots honor robots.txt directives in its site-owner guidance.
Treat Google Search and Google-Extended separately
Google uses Googlebot crawl controls for Search, including AI Overviews and AI Mode. To qualify as a supporting link in those experiences, a page must be indexed and eligible to appear in Search with a snippet, according to Google Search Central.
Google-Extended is a robots.txt control token, not a separate HTTP user-agent string. Google says it controls whether crawled content may be used for:
- Training future Gemini models powering Gemini Apps and the Vertex AI API for Gemini.
- Grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Here, grounding means providing content from the Search index to the model at prompt time.
Google says this token does not affect inclusion in Google Search and is not a Search ranking signal. These boundaries are specified in Google's Google-Extended documentation.
The decision is therefore whether to permit that combined training-and-grounding scope. Blocking it while allowing Google Search is not the same as preserving every Google AI retrieval use.
Translate the policy into robots.txt rules
Use the examples below as configuration patterns to review with your site administrator, not as a replacement robots.txt file. Preserve existing path restrictions, sitemap declarations and other crawler policies.
Before adding explicit bot groups, check how they interact with existing exclusions. Google's robots.txt specification says its crawlers select the most specific matching user-agent group. They do not combine that group's rules with the wildcard group. Ask your administrator to check which exclusions need to appear in each relevant group rather than assuming they carry over.
Allow search and restrict training collection
For content intended to be fully crawlable by these search bots, this pattern reflects the separate controls documented by OpenAI and Anthropic:
# Search access for public content
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Restrict training collection
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Do not copy the full-site allow rules if your policy excludes particular paths. Have your administrator adapt the groups to the intended public-content boundary.
Decide on Google-Extended separately
If your policy restricts both the Gemini training and grounding uses described above, apply Google's documented control token:
# Restrict the specified Gemini training and grounding uses
User-agent: Google-Extended
Disallow: /
Review Googlebot access separately. Do not label this configuration as blocking training only.
Also check each relevant subdomain. Google's specification limits a robots.txt file's scope to its host, protocol and port, as explained in its file-location and validity rules.
Make a separate decision about user-requested visits
A user asking an assistant to open a page is not the same as automatic search crawling.
OpenAI says ChatGPT-User handles certain user-triggered visits, is not an automatic web crawler and may not be subject to robots.txt rules. It does not control Search eligibility. See OpenAI's user-agent guidance.
Anthropic documents a different boundary: blocking Claude-User prevents retrieval in response to user queries and may reduce user-directed search visibility. Its crawler guidance includes robots.txt controls for that bot.
Record your preference for user-requested retrieval explicitly. A training-crawler rule does not settle this separate decision.
Check deployed access, not just the policy file
Ask your administrator to review hosting, firewall and CDN access alongside robots.txt. OpenAI recommends allowing its published searchbot IP ranges as well as OAI-SearchBot; the ranges are linked in its crawler documentation.
Access still does not guarantee visibility. Google explicitly says meeting its requirements does not guarantee crawling, indexing or serving, and includes CDN and hosting access in its AI features guidance.
Give your administrator a provider-by-provider policy covering search, training, grounding and user-requested visits. Before changing a blanket “block AI” setting, compare its effects with the official bot definitions and review the deployed rules against the uses you actually want to permit.
