How to Verify Search Engine Bots

A search crawler is software that visits pages to help build a search index. It may identify itself in the User-Agent header, extra information attached to its web request. That name is a claim, not proof. Before you allowlist a search bot, exempt it from a security control, or respond differently to it, verify the source IP address using the search operator's current documented method.

AI-assisted search services can use separate search, training, and user-triggered agents. See AI search bot verification for OAI-SearchBot, PerplexityBot, Claude-SearchBot, Amzn-SearchBot, DuckAssistBot, and MistralAI-Index.

How can I verify a search bot request?

Find the source IP address that connected to infrastructure you control. A content delivery network (CDN) serves content through distributed servers, and a reverse proxy receives requests on your website's behalf. If a trusted service accepted the visitor's connection before forwarding the request to your server (the origin), use only a client-address field that your origin accepts exclusively from that intermediary. Ensure that the intermediary removes or overwrites client-supplied copies of the field, restrict direct access to the origin where possible, and record the address of the system that connected directly to your server (the immediate peer) separately. Never trust a forwarding header received directly from an untrusted client.
Record the request time and complete User-Agent value. Use the crawler name only to select the operator documentation; do not use it as authentication. When verification affects a security decision, also record the method, operator, claimed role, result, verification time, and evidence. For an IP-range check, retain the feed URL and a version identifier such as its ETag, Last-Modified value, or content hash. For forward-confirmed reverse DNS (FCrDNS), retain the accepted hostname—a DNS name for a host or service—from the Pointer (PTR) record and the matching forward address.
Follow the operator's official method. This is usually a match against current published IP ranges, forward-confirmed reverse DNS (FCrDNS), or an operator-provided verification tool.
Treat a request as verified only when the source address passes that method. If the check fails or cannot be completed, leave the request unverified rather than assuming that it is either genuine or malicious.

An IP address tag, registered holder, Autonomous System Number (ASN), plausible hostname, or matching User-Agent can support an investigation. None of them authenticates a crawler by itself. Passing an operator's check establishes that the connection used an address or DNS identity the operator documents for that traffic category. It does not prove the exact software that sent the request, its purpose, that its behavior is harmless, or that it is entitled to bypass authentication, authorization, input validation, or every security control. When an address list or DNS namespace is shared by several crawler roles, the source category can be verified while the exact User-Agent role remains self-declared.

Scope any exemption narrowly. A verified crawler might bypass a browser challenge that it cannot complete, while remaining subject to authorization checks, request validation, sensible resource limits, and monitoring.

robots.txt expresses crawl preferences to compliant crawlers. It is not authentication, authorization, access control, or a confidentiality mechanism. It is publicly readable and does not stop a noncompliant client from requesting a URL. Protect non-public content with application authentication and network controls.

How does forward-confirmed reverse DNS verify a crawler?

A Pointer (PTR) record associates an IP address with a hostname. First, look up the PTR record for the source IP address. If more than one hostname is returned, evaluate each one separately. DNS names are case-insensitive; remove a final root-label dot before comparison. Match a hostname only against the operator's exact documented pattern and at DNS label boundaries. For an approved *.example.com pattern, crawler.example.com matches, while notexample.com, bare example.com, and example.com.attacker.test do not.

Then forward-resolve that same hostname. For an original IPv4 address, one of the returned A records must equal that address. For IPv6, one of the AAAA records must equal it. A PTR result without this forward confirmation is not enough. A timeout, SERVFAIL, malformed response, or absence of a matching forward address leaves the request unverified. See Reverse DNS for command examples and the limits of PTR records.

How do published IP ranges verify a crawler?

Some operators publish crawler addresses as CIDR prefixes or through a verification service. Compare the source IP address with the current operator-published ranges for the claimed crawler. Use the correct list when an operator separates search crawlers from special-purpose or user-triggered fetchers.

Fetch and refresh machine-readable lists according to the operator's instructions and HTTP caching headers. Do not copy the addresses shown today into a long-lived rule or documentation page: crawler infrastructure and published ranges can change. Retrieve a feed from its exact official HTTPS URL, and do not follow a redirect to an unrelated host.

Validate the response status, media type, schema, and every address or CIDR before replacing the active set. Reject unsafe entries such as 0.0.0.0/0 or ::/0. Quarantine unexpectedly large changes in entry count or prefix breadth for review, then apply the complete validated set atomically.

If retrieval, parsing, or validation fails, retain the last-known-good set for a bounded grace period and alert; never install an empty or partially parsed set. Track the last successful HTTP retrieval or cache revalidation. Once the configured maximum staleness is exceeded, stop granting crawler-specific exemptions and process matching requests normally as unverified. Retain the stale set only for diagnostics.

How can I verify common search bots?

This table is representative rather than exhaustive.

Crawler	What it does	Official verification guidance
Googlebot	Crawls pages and resources for Google Search using mobile and desktop variants.	Use Google's current common-crawlers.json feed, or use FCrDNS with the documented googlebot.com or geo.googlebot.com suffix. These methods attribute the source to Google's common-crawler category, which includes several crawler identities; the exact User-Agent role remains self-declared.
Bingbot	Bing's standard crawler discovers and renders content for its search index.	Use Bing's verification tool, current Bingbot IP-range file, or FCrDNS with the search.msn.com suffix.
Applebot	Crawls content for search features across Apple's ecosystem, including Spotlight, Siri, and Safari.	Match Apple's current Applebot CIDR file, or use FCrDNS with the applebot.apple.com suffix.
DuckDuckBot	Crawls the web to improve DuckDuckGo search results.	Match the source address against DuckDuckGo's current published list or machine-readable JSON feed. Do not preserve a copied snapshot as a permanent allowlist.
YandexBot	Is Yandex's main crawler for indexing web pages.	Use Yandex's IP-address check tool or FCrDNS with an exact yandex.com, yandex.net, or yandex.ru suffix. This attributes the source to Yandex crawler infrastructure; the exact User-Agent role remains self-declared.
Baiduspider	Crawls pages for Baidu's search index and uses related variants for specialized search products.	Baidu documents reverse hostnames under baidu.com or baidu.jp. Forward-resolve the exact hostname and confirm the original IP address before trusting it.
Yahoo Slurp	Crawls for Yahoo Search, Yahoo Mobile Search, and content used by other Yahoo services.	Yahoo documents the Slurp User-Agent but does not publish a stronger IP-range or DNS authentication method on its crawler page. Treat the User-Agent as an unverified claim.
Brave Search crawler	Discovers pages for Brave Search's independent index.	Brave says its crawler does not advertise a differentiated User-Agent. Its public guidance does not provide a separate Bravebot identity that a site can authenticate from request logs.

What should I avoid when trusting a search bot?

Do not trust a crawler name or documentation URL embedded in a User-Agent header.
Do not accept a generic forward and reverse DNS match without the operator's exact suffix.
Do not substitute IP ownership, ASN, location, or an IP Lens tag for operator verification.
Do not rely on a stale copy of a published range list when the operator provides a current feed.
Do not verify the proxy or CDN address when a trusted intermediary has supplied the original client address through a correctly configured trust boundary. Do not trust a client-supplied forwarding header merely because it is present.