2026-09-17, GEO · Engineering

AI crawlers explained: OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot, Google-Extended and more

An AI crawler is an automated client an AI company uses to fetch web pages, either in bulk (to build a search index or a training corpus) or on demand (when a user’s question needs a page). Most vendors now split these jobs across separate user agents, which means robots.txt can express “appear in AI search, but don’t train on my content”. It also means a single careless Disallow can remove you from an engine you wanted to be in.

Below: the user agents each major vendor documents, the stated purpose, robots.txt behavior and the stated effect of blocking, all from vendor pages checked 2026-09-17. Then robots.txt recipes and a way to measure the effect of a change.

The crawler table

Sources: OpenAI crawlers, Perplexity crawlers, Anthropic crawler article (dated April 7, 2026), Google common crawlers, About Applebot, Bing crawlers. All checked 2026-09-17.

xAI is not in the table. We found no first-party xAI page documenting Grok’s crawler user agents as of 2026-09-17. Third-party lists name several different tokens and disagree with each other, so we don’t repeat any of them here.

What each vendor says about visibility

OpenAI

• OAI-SearchBot: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” OpenAI recommends allowing it in robots.txt and allowing requests from its published IP ranges.
• The settings are independent: “a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot”. If both are allowed, OpenAI “may use the results from just one crawl for both use cases”.
• Timing: “it can take ~24 hours from a site’s robots.txt update for our systems to adjust” for search.
• ChatGPT-User “is not used to determine whether content may appear in Search”. OpenAI points to OAI-SearchBot for search opt-outs.
• The Publishers and Developers FAQ adds that if OpenAI obtains a disallowed page’s URL from a third-party search provider or by crawling other pages, it may surface just the link and title in ChatGPT Atlas, and that noindex prevents this, provided the crawler is allowed to read the page.

Perplexity

• “To ensure your site appears in search results, we recommend allowing PerplexityBot in your site’s robots.txt file and permitting requests from our published IP ranges.”
• Changes “may take up to 24 hours” to be reflected.
• Perplexity-User may visit a page to answer a question “and include a link to the page in its response”. Since the fetch is user-requested, it “generally ignores robots.txt rules”.
• Perplexity’s page also includes WAF guidance: if a firewall blocks its IP ranges, robots.txt permission alone won’t let the bot in.

Anthropic

• Restricting ClaudeBot “signals that the site’s future materials should be excluded from our AI model training datasets.”
• Disabling Claude-SearchBot “may reduce your site’s visibility and accuracy in user search results.”
• Disabling Claude-User “prevents our system from retrieving your content in response to a user query, which may reduce your site’s visibility for user-directed web search.”
• Anthropic states its bots honor robots.txt, support the non-standard Crawl-delay extension, and that blocking by IP address “may not work correctly or persistently guarantee an opt-out”.

Google

• AI Overviews and AI Mode: AI features and your website says “robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search”, and that nosnippet, data-nosnippet, max-snippet and noindex limit what’s shown.
• Google-Extended: manages whether crawled content “may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding … in Gemini Apps and Grounding with Google Search on Vertex AI.” It “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”

So disallowing Google-Extended affects more than training: by Google’s description, it also opts content out of grounding in the Gemini app.

Apple

• Enabling Applebot “allows website content to appear in search results for Apple users” in Spotlight, Siri and Safari. Applebot data may also train Apple foundation models and provide context for AI-generated answers.
• Disallowing Applebot-Extended opts out of training Apple’s foundation models; “webpages that disallow Applebot-Extended can still be included in search results.”
• To keep content out of AI-generated “broad world knowledge answers”, Apple points to the nosnippet meta tag, not robots.txt.
• If robots.txt doesn’t mention Applebot but mentions Googlebot, “the Apple robot will follow Googlebot instructions.”

Microsoft

• Bing says robots.txt controls how its crawlers interact with a site.
• For AI answers, a September 2023 Bing Webmaster blog post, written for what was then Bing Chat, says content with NOCACHE may be included in answers with only URL, title and snippet, content with NOARCHIVE will not be included in answers, and both still appear in Bing search results.
• Bing Webmaster Tools’ AI Performance report (February 10, 2026) shows citations across Microsoft Copilot and AI summaries in Bing, and states that “Bing respects all content owner preferences expressed through robots.txt and other supported control mechanisms.”

robots.txt recipes

Robots.txt groups match by user agent token; a crawler follows the most specific group that names it and ignores the rest. Test any change on a staging copy first, and keep your existing User-agent: * group intact.

Recipe 1: appear in AI search, opt out of model training

# Search and answer crawlers: allowed
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

# Training crawlers and tokens: disallowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
Disallow: /

This leaves Googlebot, Applebot and bingbot to your existing rules. Two tokens are left out on purpose:

• Google-Extended is not in the disallow group. Blocking it would also remove your content from grounding in the Gemini app, per Google. Add it only if you accept that trade-off.
• ChatGPT-User and Perplexity-User are not listed. Their vendors say robots.txt may not apply to them, so a rule gives a false sense of control.

Recipe 2: opt out of everything AI, keep classic search

User-agent: OAI-SearchBot
User-agent: GPTBot
User-agent: PerplexityBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

This does not remove you from Google AI Overviews or AI Mode, which Google controls through Googlebot and snippet directives, and it does not cover Apple’s AI answers, which Apple controls through nosnippet. Expect ChatGPT to still show bare navigational links in some cases, as OpenAI describes.

Recipe 3: keep a private section out of every AI crawler

User-agent: OAI-SearchBot
User-agent: GPTBot
User-agent: PerplexityBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /internal/
Allow: /

Anything that must stay private belongs behind authentication. robots.txt is a public request that compliant crawlers honor; it doesn’t restrict access.

Verify the crawler before you trust the user agent

User agent strings are easy to fake. Vendors publish ways to check them:

• OpenAI: searchbot.json, gptbot.json, chatgpt-user.json
• Perplexity: perplexitybot.json, perplexity-user.json
• Google: the common crawlers’ IP list and reverse DNS under googlebot.com
• Apple: reverse DNS under applebot.apple.com and a published CIDR list
• Bing: the Verify Bingbot tool

Before blaming a vendor for ignoring your rules, match the requesting IP against its list. Before blaming robots.txt for a visibility drop, check your WAF and CDN logs for blocked requests from those ranges.

Measuring the effect of a crawler change

The vendor statements tell you the direction of an effect, for example that opting out of OAI-SearchBot removes you from ChatGPT search answers. They don’t give the size of the effect on your prompts or how long the transition takes.

Design

1. Fix a prompt set of 30–100 prompts where your domain is plausibly citable: category, comparison and how-to questions in your space. Prompt set design covers selection.
2. Pick the treated engine and a control. If you change only OAI-SearchBot, ChatGPT is treated. Gemini and Copilot are controls, because the change doesn’t touch Google or Bing.
3. Record a baseline for at least a week, with several runs per prompt, because answers vary between runs.
4. Change robots.txt and note the timestamp. Wait at least the vendor’s stated propagation time (about 24 hours for OpenAI and Perplexity).
5. Measure for the same length of time afterwards.
6. Compare the change in citation rate on the treated engine with the change on the controls. A drop on ChatGPT alone points to the change; a drop everywhere points to something else.

Metric

Citation rate = answers with at least one sources[] URL on your domain ÷ answers collected, per engine and period. Track mention rate (brand name in text) separately; a crawler change affects citations first. See mentions vs citations.

Code

import os
from urllib.parse import urlsplit

import requests

API = "https://api.answerline.dev"
HEADERS = {"Authorization": f"Bearer {os.environ['ANSWERLINE_API_KEY']}"}
DOMAIN = "example.com"


def cites(result: dict, domain: str) -> bool:
    for source in result.get("sources", []):
        host = (urlsplit(source.get("url", "")).hostname or "").lower()
        if host == domain or host.endswith("." + domain):
            return True
    return False


def submit_run(prompts: list[str], engines: list[str], run_id: str, hook: str) -> None:
    tasks = [
        {
            "taskType": engine,
            "payload": {"prompt": prompt, "country": "US"},
            "idempotencyKey": f"crawler-test-{run_id}-{engine}-{i}",
            "webhook": {"url": hook},
        }
        for engine in engines
        for i, prompt in enumerate(prompts)
    ]
    for start in range(0, len(tasks), 500):
        resp = requests.post(f"{API}/v1/async/task/batch", json=tasks[start:start + 500], headers=HEADERS, timeout=60)
        resp.raise_for_status()


def citation_rate(results: list[dict]) -> float:
    return sum(cites(r, DOMAIN) for r in results) / len(results) if results else 0.0

Your webhook handler stores each response.result with the engine, prompt index, run id and timestamp. Compute citation_rate per engine for the before and after periods, then the difference-in-differences: (treated after − treated before) − (control after − control before).

How many runs

If ChatGPT cites you in 30% of answers before the change, detecting a drop to 20% at 80% power needs roughly 290 answers per period. With 50 prompts that is about six runs per prompt per period. Smaller effects need far more. Because runs of the same prompt are correlated, also compare the per-prompt rates before and after, not just the pooled rate.

Cost

Async tasks cost 5 credits on ChatGPT, 4 on Gemini and 5 on Copilot. 50 prompts × 6 runs × those three engines is 300 × 14 = 4,200 credits per period, 8,400 for before and after. Synchronous calls add 2 credits each. See credits and pricing.

Related: the AI search engines and how they cite, llms.txt, and the AI visibility tracking use case. To start measuring, make a first call with the quickstart.

Keep reading