Blog / seo wednesday
What your server logs show about Googlebot
Your access log records every request Googlebot makes to your site. Here is how to open it in cPanel, read it, and act on what it tells you.
Search Console tells you what Google decided about your site. Your access log tells you what Google actually did: every request, the exact URL, the response code, the time. When indexing goes sideways and Search Console only gives you a vague category, the specific answer is usually sitting in the log already. This week's SEO Wednesday is about reading it, which takes maybe twenty minutes to learn and is one of the few diagnostics nobody can sell you.
Finding the log in cPanel
In cPanel, open Metrics, then Raw Access. You will see a downloadable log for each domain on the account, and on our hosting it is there by default with nothing to switch on. Before you do anything else, tick the archiving options on that page so logs are kept at the end of each month instead of being discarded on rotation. Logs rotate on a cycle, and the log you want is nearly always last week's.
Download the .gz file and unzip it. Every line is one request, in combined log format:
66.249.66.1 - - [21/Oct/2026:04:12:55 -0400] "GET /services/ HTTP/1.1" 200 18422 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"Reading left to right: the IP, the timestamp, the method and URL requested, the status code we returned, the bytes sent, the referrer, and the user agent. That is the whole format. If you are comfortable at a shell, cPanel's Terminal works fine for this. If not, a text editor that can search a large file will get you most of the way.
Four counts worth running
Start with how Google spends its time on your site:
grep Googlebot access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -40That gives you the forty URLs Googlebot requested most. If the top of that list is your real pages, good. If it is full of ?replytocom= links, feed URLs, internal search results, or tag archives with one post in them, Google is spending its crawl on pages you would never want ranked.
Next, the errors Google keeps hitting:
grep Googlebot access.log | awk '$9 >= 400 {print $9, $7}' | sort | uniq -c | sort -rnA handful of 404s is normal. The same 404 requested fifty times means something is still linking to it, often your own sitemap or an old menu item. We cover the fix in broken links and 404s.
Third, redirect chains. Swap the condition to $9 == 301 and look for URLs that redirect to something that also redirects. Chains waste a crawl and slow real visitors down too. Redirects done right has the patterns we use.
Fourth, impostors. Plenty of scrapers set Googlebot as their user agent. Check an IP with a reverse lookup, then a forward lookup on the name you get back, which is the method Google documents. Real Googlebot resolves to googlebot.com or google.com and the forward lookup returns the same IP. Anything else is just traffic, and if it is hammering you, it belongs in a firewall rule rather than in your SEO thinking.
Where Search Console still wins
Logs beat Search Console on per-URL detail. Search Console beats logs on availability history. Its Crawl Stats report, under Settings then Crawl stats, shows ninety days of host status for robots.txt fetching, DNS resolution and server connectivity. It also splits requests by response code and by file type, and separates discovery of new URLs from refreshes of pages Google already knows. Google notes the report is aimed at advanced users, that it only appears for root-level properties, and that it is generally unnecessary below about a thousand pages. Use it to spot a bad week, then go to the log for the specific URLs.
Turning a log read into changes
Three or four things usually come out of a first read. Junk URL patterns getting crawled: block them in robots.txt or stop generating the links. Repeatedly crawled 404s: restore the page or redirect it to the closest real equivalent, one hop. Slow responses: look at the bytes column and the pages where it balloons, because a page that is slow for Googlebot was slow for a person first. And occasionally the log shows a clean, boring crawl of exactly your real pages, which is a result worth having, because it means the problem is content or links rather than anything technical.
One warning. A log read is a snapshot of a week, not a trend, so do not rewrite your whole site off a single file. Compare two months before you conclude anything about direction.
If you have pulled a log and cannot tell what you are looking at, or the Raw Access page is not giving you what you expect, open a ticket with our support team and send us the file. We read these every day and we will tell you plainly what we see in it.