Blog / seo wednesday
XML Sitemaps: What Actually Matters for Small Sites
A plain guide to XML sitemaps for small sites: what to include, which fields search engines ignore, and the server problems that break them.
Most site owners we talk to have a sitemap and have never opened it. That is usually fine, because the SEO plugin generated one and it works. But when pages stop showing up in search, the sitemap is one of the first places worth looking, and a surprising number of them are pointing at pages that no longer exist or were never meant to be public. Every SEO Wednesday we pick something you can finish in an afternoon. This one takes about twenty minutes.
What a sitemap does, and what it does not do
A sitemap is a list of URLs you would like search engines to know about. That is the whole job. It is a hint, not an instruction, and submitting a URL does not oblige anyone to index it. We have seen owners resubmit a sitemap five times waiting for a thin page to appear, when the page was simply not worth indexing in the crawler's judgment.
Where sitemaps genuinely help: brand new sites with almost no inbound links, large catalogues where some products sit five clicks deep, and pages that nothing else on the site links to. Where they help least: a well-linked 30-page business site. If your navigation and internal links are sound, crawlers will find everything anyway. Our older post on internal linking for small sites covers that side of it.
Put only indexable pages in it
This is the rule that gets broken most often. A sitemap should contain the canonical, self-referencing, 200-status version of pages you want in search results. Nothing else.
That means no redirected URLs, no 404s, no pages carrying a noindex tag, no URLs blocked in robots.txt, and no http:// entries if your site runs on https. Mixing those in sends a contradictory signal: you have asked for a page to be indexed and told the crawler to ignore it in the same breath. It will not earn a penalty, but it wastes attention and it makes the coverage reports much harder to read.
Two extras worth checking on WordPress. First, tag and author archives often end up in the sitemap by default, and on a small site those are usually thin pages nobody searches for. Turn them off in your SEO plugin unless you actively use them. Second, if you have a staging or development copy on a subdomain, make sure it is not generating a public sitemap of its own. We have cleaned up more than one site where the staging copy was indexed alongside the live one.
The fields nobody needs to argue about
The sitemap format allows four elements per URL: loc, lastmod, changefreq and priority. Google has said publicly that it ignores changefreq and priority entirely, so tuning those numbers is time you will not get back. Set every page to priority 1.0 if you like. It changes nothing.
lastmod is the one that still counts, and only if it is honest. If your CMS stamps today's date on every URL every time the sitemap regenerates, the value becomes noise and gets discounted. If it reflects the date the content actually changed, it is a useful signal for recrawling. Most modern plugins handle this correctly. It is worth spot-checking after a theme change or a bulk edit, because a find-and-replace across 200 posts can touch every modified date at once.
Size limits are generous: 50,000 URLs or 50MB uncompressed per file, whichever comes first. Past that you split into multiple sitemaps and reference them from a sitemap index file. Almost no small site gets near this.
Submit it once and then watch the report
Add the sitemap URL in Google Search Console under Sitemaps, and in Bing Webmaster Tools if you want the coverage. Add a Sitemap: line to your robots.txt too, since other crawlers read it there. Then stop resubmitting. The file gets refetched on its own schedule, and pressing the button again does not speed anything up.
What is worth your time is the report that comes back. Discovered URLs versus indexed URLs, and the reasons given for anything excluded, will tell you more about your site's health than the sitemap itself ever will. We wrote about reading that report in the Search Console indexing report.
The server-side failures we actually see
A few sitemap problems are ours rather than yours, and they are quick to rule out. Open the sitemap URL in a private browser window and confirm it returns XML, not a 404 or a login page. Check that a security plugin or firewall rule is not blocking crawler user agents from fetching it, which happens more than you would expect after a security cleanup. Watch for aggressive full-page caching serving a sitemap from three months ago. And if fetches are failing intermittently with a server error, the sitemap is a symptom, not the problem; our status page is the first place to check when a site behaves oddly at odd hours.
If your sitemap looks wrong and you cannot tell whether it is the plugin or the server, send us the URL and we will look at the request logs with you. Open a ticket with support and we will tell you what our end is actually returning to the crawlers.