Why look at crawler logs
Search Console tells you what Google chose to report; your server logs tell you what Googlebot actually requested, every time. Logs answer questions no other source can:
- Is Googlebot spending its visits on pages that matter, or on endless filter and parameter URLs?
- Which URLs return 404 or 5xx to crawlers, and how often?
- How quickly does Googlebot pick up new or changed sections?
- How much do AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) take, and do they respect your robots.txt?
- Are there "Googlebots" that are really scrapers pretending to be Google?
How to get your logs
- Nginx: usually
/var/log/nginx/access.log(older ones rotated toaccess.log.1,access.log.2.gz...). - Apache:
/var/log/apache2/access.logor/var/log/httpd/access_log. - Shared hosting: cPanel's "Raw Access" section, or your host's control panel.
- CDNs (Cloudflare, Fastly, CloudFront) keep their own logs in their own formats; this tool reads the combined format only, so export or convert those first.
The tool expects the standard combined format, %h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i", which is the default for Nginx and common for Apache. Lines with a virtual-host prefix are also understood. Without user agents (the common format) bots cannot be recognised.
Verifying Googlebot
Anyone can send the user agent "Googlebot". The tool checks each claimed Google crawler's IP address against the list of IP ranges Google publishes for its common crawlers (a copy is bundled with this page; the date is shown with the results). Requests from other addresses are counted separately as unverified and left out of the Google figures. Google also documents a reverse-DNS check (the address resolves to a googlebot.com or google.com host, which resolves back to the same address), which a browser cannot perform.
Reading the results
- Hits per day: sudden drops often follow server errors or robots.txt changes; sudden spikes may follow a site change Google is re-crawling, or an infinite URL space it has discovered.
- Status codes: a few percent of 3xx and 404 is normal on any real site. A steady share of 5xx is not: Google slows down crawling when a server errors or responds slowly.
- Parameter URLs: a high share means crawlers spend time on sort orders, filters, session IDs or tracking parameters. Consider robots.txt rules for parameter patterns and consistent internal links to clean URLs.
- Top 404s: fix the internal links pointing to them, or redirect pages that moved. 404s for pages that really are gone are fine.
- Resource types: Googlebot also fetches CSS, JavaScript and images to render pages; that is expected and healthy.
Privacy
Access logs contain visitors' IP addresses, which are personal data in many jurisdictions. This page reads files with the browser's File API and processes them in memory on your device. Nothing is uploaded; you can disconnect from the internet after the page has loaded and it still works.