How to Verify AI Search Bots

AI search services use crawlers—programs that visit web pages—to build indexes and fetch sources for answers. A bot may identify itself in the User-Agent header, extra information attached to its web request. That name is only a claim. Verify the source before classifying the traffic as genuine or granting a narrowly scoped exception to a bot challenge, rate limit, or similar control. Verification can confirm that a request came from an operator's documented network or passed its signature check. What this proves depends on the method. It does not make a request safe or authorize access to non-public content.

This guide covers AI search crawlers, indexing bots, and user-triggered web fetchers. For general-purpose search crawlers such as Googlebot, bingbot, Applebot, and DuckDuckBot, see Search bot verification.

How can I verify an AI search bot request?

Record the address of the system that connected directly to your server (the transport peer). A reverse proxy receives requests on your website's behalf; a content delivery network (CDN) serves content through distributed servers. If one of these trusted services accepted the visitor's connection, derive the real client IP only from information that intermediary supplied after removing or replacing untrusted client values (sanitizing them). Do not accept an arbitrary Forwarded or X-Forwarded-For value.
Record the request time in UTC, complete User-Agent value, transport peer address, derived client address, and the method used to derive it. Retain any request-signature headers needed for later verification.
Map the claimed User-Agent to the exact documented role: automatic search or indexing, training, general-purpose crawling, or user-triggered retrieval.
Apply the operator's current verification method. Normally this means requiring both the expected User-Agent and a source address in the corresponding official feed. When an operator documents forward-confirmed reverse DNS or request signatures, follow that procedure instead of inventing a substitute.
If the operator publishes no verification method, the feed is unavailable beyond your permitted staleness period, or any required check fails, leave the request unverified.
Record the evidence used for the decision, including the matched prefix, feed retrieval time, feed version or content hash, and signature result where applicable.

An IP Lens tag, registered holder, Autonomous System Number (ASN), location, expected User-Agent, or plausible reverse DNS can support an investigation. None authenticates an AI crawler by itself. Require the operator's documented combination of signals. An address match verifies only what the operator says that feed represents.

A hostname is a DNS name for a host or service. A Pointer (PTR) record associates an IP address with a hostname, but a plausible-looking name by itself is not verification. Forward-confirmed reverse DNS can be valid when the operator documents the required suffixes and forward lookup procedure, as general search operators such as Google, Bing, and Apple do. See Search bot verification.

AI crawler verification assurance levels
Classification	Meaning
Unverified claim	The request presents a bot User-Agent, but no current operator-published verification method succeeded.
Operator-network verified	The source matches a shared operator range, but the exact role is still identified only by the User-Agent.
Role/network matched	The expected User-Agent and the operator's corresponding role-specific network feed both match. This is not cryptographic proof of request intent.
Cryptographically authenticated	A valid supported request signature authenticates the documented bot identity.

Apply the narrowest useful exception. A verified crawler may bypass an interactive bot challenge on public pages while remaining subject to authentication, path and method restrictions, request-size limits, concurrency controls, and abuse monitoring.

Fetch each feed over HTTPS with strict timeouts and a bounded response size. Validate the operator-specific schema and every CIDR before replacing the active set, then install the complete new set atomically. If retrieval, parsing, or validation fails, keep the last-known-good set temporarily and alert. Track its age and stop using it for privileged policy exceptions after a configured hard-expiry period rather than trusting stale ranges indefinitely.

A current feed verifies current membership, not necessarily historical membership. For later audits, retain timestamped feed snapshots or at least the retrieval time, content hash, matched prefix, and source document version.

robots.txt is a voluntary instruction for cooperating crawlers, not authentication or access control. It is publicly readable and does not prevent a noncompliant client from requesting a URL. Protect non-public content with application authentication and network controls.

How can I verify common AI search bots?

Common AI crawler roles and verification methods
Crawler	What it does	How to verify and control it
`OAI-SearchBot`	Crawls pages so they can be surfaced in ChatGPT search results.	Require the expected `OAI-SearchBot` User-Agent and a source address in OpenAI's current search-bot range feed. Its robots.txt control is independent of `GPTBot`, which crawls content that may be used to train OpenAI's generative AI foundation models.
`PerplexityBot`	Crawls pages so they can be surfaced and linked in Perplexity search results.	Require the expected `PerplexityBot` User-Agent and a source address in its current range feed. Keep it separate from `Perplexity-User`, which fetches pages for user requests.
`Claude-SearchBot`	Indexes content to improve search result quality for Claude users.	Require the expected `Claude-SearchBot` User-Agent and match the source against Anthropic's current `bots.json` list. The list is shared across Anthropic crawler roles, so the address match verifies Anthropic origin but does not independently authenticate the exact bot role.
`Amzn-SearchBot`	Crawls content so it can be eligible for Amazon search experiences such as Alexa.	Require the expected `Amzn-SearchBot` User-Agent and a source address in Amazon's current search-bot range feed. Keep it separate from `Amazonbot`, which may support model training, and `Amzn-User`, which performs user-triggered live retrieval.
`DuckAssistBot`	Fetches pages in real time for DuckDuckGo's AI-assisted answers.	Require the expected `DuckAssistBot` User-Agent and a source address in DuckDuckGo's current list. DuckDuckGo says this data is not used to train AI models and that its control is separate from organic search inclusion.
`MistralAI-Index`	Automatically indexes content for Mistral search, which helps answer questions in Vibe.	Require the expected `MistralAI-Index` User-Agent and a source address in its current range feed. Mistral documents separate agents for training and user-triggered retrieval.

How are search, training, and user-triggered agents different?

AI crawler and fetcher roles by operator
Operator	Search or indexing	Training	User-triggered retrieval
OpenAI	`OAI-SearchBot`	`GPTBot`	`ChatGPT-User`
Amazon	`Amzn-SearchBot`	`Amazonbot`	`Amzn-User`
Google	`Googlebot`	`Google-Extended` control token, not a crawler	`Google-Agent`
Perplexity	`PerplexityBot`	No separate training crawler is documented; neither documented agent collects content for training foundation models	`Perplexity-User`
Anthropic	`Claude-SearchBot`	`ClaudeBot`	`Claude-User`
Mistral	`MistralAI-Index`	`MistralAI-Training`	`MistralAI-User`
DuckDuckGo	`DuckAssistBot`	`DuckAssistBot` data is not used for training	No separate agent documented on this crawler page

Configure each documented token for the outcome you want. Allowing search indexing does not require allowing model training. User-triggered fetchers also differ: OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Amazon says Amzn-User may not follow every directive. Do not assume one company-wide rule covers every agent.

What about general search crawlers and control tokens?

Verify Googlebot, bingbot, Applebot, and DuckDuckBot using their established methods in the traditional search bot guide. Brave is different: Brave says its search crawler does not advertise a differentiated User-Agent, and its documentation publishes no request-level IP, reverse-DNS, or signature verification method. Do not classify a request as verified Brave crawler traffic solely from its User-Agent, ASN, or apparent network ownership.

Google-Extended is a robots.txt product token, not a separate User-Agent sent in HTTP requests. It controls whether content crawled with existing Google agents may be used for Gemini model training and grounding. It does not control inclusion or ranking in Google Search.

Google-Agent is different: it is a user-triggered fetcher used by agents hosted on Google infrastructure and uses Google's user-triggered-agents.json ranges. Google also signs a subset of Google-Agent requests with the experimental Web Bot Auth protocol. A successfully verified signature authenticates the documented https://agent.bot.goog identity; it does not authorize the requested operation. Not every request is signed, so use Google's established IP-range, reverse-DNS, and User-Agent checks as the fallback.

Applebot-Extended also does not crawl pages. It controls whether data collected by Applebot may be used to train Apple's foundation models. Disallowing it does not block Applebot or prevent content from appearing in Apple's search features. Use noindex or nosnippet in a robots meta tag or an X-Robots-Tag response header. These are indexing directives, not robots.txt rules. noindex prevents indexing, while nosnippet prevents a description or web answer but still allows a title-only suggestion.

What should I avoid when trusting an AI search bot?

Do not trust a crawler name or documentation URL embedded in a User-Agent header.
Do not use a search bot's range feed to authenticate a separate training or user-fetch agent.
Do not infer an exact crawler role from a shared operator range feed.
Do not substitute IP ownership, ASN, location, reverse DNS, or an IP Lens tag for verification.
Do not keep a copied range list as a permanent allowlist when the operator publishes a current feed.
Do not use a current range feed as proof of historical membership without retained evidence.
Do not confuse a robots.txt control token with a User-Agent that will appear in request logs.
Do not verify a proxy or CDN address when the authenticated real client address is available.
Do not turn verified identity into a broad bypass of authentication or other application controls.