Blog / seo wednesday
How to find and fix broken links on your site
Broken links and 404 pages pile up on every site. Here is how we find them from logs and Search Console, and how we decide which ones deserve a redirect.
Every site collects dead links. You rename a page, a plugin changes its permalinks, a product gets discontinued, somebody links to you with a typo in the URL. None of that is a crisis on its own. But a site carrying a few hundred dead URLs wastes crawl time, and a visitor who lands on a blank error page usually leaves instead of hunting for the right one. For this week's SEO Wednesday, here is the way we go looking for them, and how we decide which ones are worth the effort.
Pull the list you already have
Search Console's Page indexing report is the first stop, because it shows you the dead URLs Google has actually seen. Open it and look at two rows: "Not found (404)" and "Soft 404". Click into either and you get the URL list plus the date each one was last crawled. Export it. (We walked through the rest of that report in reading the indexing report.)
Your access log is the second source, and the more useful one. In cPanel, Raw Access Logs under Metrics gives you the full request log. If you have shell access, this ranks your dead URLs by how often anything actually asks for them:
grep ' 404 ' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -40That tells you something Search Console will not. A broken URL requested 200 times a week has a real referrer behind it and deserves a redirect. One hit twice in a year can stay broken forever and nothing bad happens.
Crawl your own site
Google only reports what Google found. To catch broken internal links before a visitor does, spider the site yourself. On any Linux machine, including a shell account on one of our servers:
wget --spider -r -nd -nv -o crawl.log https://example.com/
grep -B1 'ERROR 404' crawl.logRead the log once it finishes. Internal broken links are the ones to fix first, because they are entirely in your hands. Nobody else has to cooperate, and every one you fix is a page that stops leaking visitors. Watch for missing images and stylesheets in that log too. They will not hurt your rankings much, but a theme asking for a font file that was deleted two redesigns ago is a request your server answers thousands of times for nothing.
Not every 404 needs a redirect
This is where most cleanup work goes sideways. The rules we use are short:
If the page moved or was replaced, 301 it to the closest real equivalent. If a product is gone and nothing has taken its place, leave the 404 alone, or return 410 Gone if your platform makes that easy. Google drops the URL from its index either way once it has confirmed the status; 410 just says you meant it.
What not to do is redirect every dead URL to your home page. Google frequently treats a redirect to an unrelated page as a soft 404, so you get no benefit, and you have made your own error reports useless. Our guide to redirects covers the mechanics of doing them properly in cPanel and in WordPress.
Soft 404s are the ones people miss
A soft 404 is a page returning 200 OK while telling the visitor there is nothing here. Empty search result pages, a category archive with every post removed from it, a "no longer available" template that is otherwise a normal page. Because the status code says success, nothing in your server log flags it. Google's crawlers judge from the content instead, and a page classed that way is not going to rank.
Two fixes, depending on the page. Either give it real content, which is usually the right answer for a thin category page, or make it return a genuine 404 or 410. Checking the status takes one command:
curl -sI https://example.com/no-such-page | head -1If that prints 200 for a URL that does not exist, you have a problem worth solving.
Make the error page do some work
The default 404 page on most themes is a dead end with a sad sentence on it. Put a search box on yours, plus links to your main sections and a way to get in touch. It costs twenty minutes and it recovers visitors who would otherwise close the tab.
One last thing, and it matters more than any of the above: a 404 is an honest answer, but a 500 or a connection timeout is not. When a server starts returning 5xx errors under crawl, Google backs off its crawl rate, and pages that were indexed can drop out while it waits. That is a hosting problem wearing an SEO costume. If your error rate climbs when traffic does, the fix is the stack underneath, not your meta tags. Our hosting plans are sized so a normal traffic spike does not turn into a server error, and the uptime history is public.
Work through the 404 list once, in order of how often each URL is requested, and you will be done in an afternoon. If you want a second pair of eyes on a redirect map, or you are not sure whether an error is coming from your site or from the server, open a ticket and we will look at the logs with you.