Skip to content
ToolMug

Crawler IP Verifier

Scrapers often pretend to be Googlebot in their user agent. The method search engines document is a reverse DNS lookup of the IP, a check that the host name belongs to their domain, and a forward lookup of that host name that must return the same IP. This tool runs those steps for you.

Processed on our server; nothing is stored

One per line, or paste log lines — IPs are picked out automatically. Up to 20 at a time.

0 IPs found

How to use

  1. Paste the IPs to check, one per line — or paste raw log lines and the IPs are picked out.
  2. Click Verify.
  3. Results marked Genuine … crawler can be trusted; ones marked Fake can be blocked at your firewall or CDN.

Options explained

Supported search engines
Google (googlebot.com, google.com, googleusercontent.com), Bing (search.msn.com), Baidu (baidu.com, baidu.jp), Yandex, Sogou (sogou.com), Apple (applebot.apple.com) and Petal (petalsearch.com).
How it checks
Reverse DNS for the IP → the host name must be one of those domains or a subdomain → forward DNS for the host name → it passes only if the original IP is returned.

Host names accepted and what each result means

The reverse DNS host name must be exactly one of these domains or a subdomain of it. A name such as crawl.googlebot.com.example.net does not count, because only the end of the name is compared.

Search engine Accepted domains
Google googlebot.com, google.com, googleusercontent.com
Bing search.msn.com
Baidu baidu.com, baidu.jp
Yandex yandex.ru, yandex.net, yandex.com
Sogou sogou.com
Apple applebot.apple.com
Petal petalsearch.com

For an IPv4 address the forward lookup asks for A records, and for IPv6 it asks for AAAA records. The IP passes only if one of them equals the address you entered. These are the results you can get in the Result column:

Result What happened
Genuine (engine) crawler The reverse name is on an accepted domain and the forward lookup returns the same IP.
Fake: host name does not resolve back to this IP The reverse name looks like a search engine, but the forward lookup does not return this IP.
Not a search engine host The IP has a reverse name, but it is not on any accepted domain.
No reverse DNS: not a known crawler The IP has no PTR record at all.
DNS lookup failed The DNS query itself failed or timed out. Try again before drawing a conclusion.
Not a valid IP The text could not be read as an IPv4 or IPv6 address.

Up to 20 IPs are checked in one go. When you paste log lines, every IPv4 or IPv6 address found in them is picked out, which includes any server address in the line as well as the visitor’s.

Frequently asked questions

Can it verify DuckDuckBot or AI crawlers like GPTBot?

Not yet. Those publish IP range lists rather than reverse DNS names; here they show as not a search engine host, which does not prove they are fake.

Where are the lookups made?

Our server queries a public DNS resolver. Only DNS lookups are made; the IPs themselves are never contacted.

Does a missing reverse DNS mean the IP is malicious?

No, only that it is not one of the listed search engine crawlers.

Privacy

This tool needs our server: what you enter or upload is sent over HTTPS, processed and then discarded, never stored. Server access logs record IP addresses and request URLs, used only to prevent abuse.

Related tools