--- title: Robots.txt mistakes that hide your pages description: How robots.txt really works, the errors we find on live sites, and the short file most small business sites should be running instead. date: 2026-09-16 updated: 2026-09-16 tags: [seo wednesday, robots txt, indexing] url: "https://drivenhost.com/blog/robots-txt-mistakes" author: DrivenHost --- A single line in a text file can keep your best page out of Google. We see it a few times a year on customer sites: traffic slides, nothing looks broken, and the cause turns out to be a robots.txt rule someone added during a redesign and never removed. It is a tiny file with an outsized ability to do quiet damage. Checking it properly takes about ten minutes, and this week's SEO post is a walk through how. ## What the file actually controls robots.txt is plain text living at the root of your site, at yoursite.com/robots.txt. It tells well behaved crawlers which paths they may request. That is the entire job. It does not protect anything, it does not remove pages from search results, and it has no effect on a person who types the URL by hand. Two details catch owners out. The file has to sit at the root of the exact host and protocol, so https://example.com/robots.txt covers that hostname only. A subdomain like shop.example.com needs its own file. The second detail is that compliance is voluntary. Google and Bing respect the rules. Scrapers read the file as a handy list of directories worth poking at, so never use it to shield a folder you would rather nobody found. ## Disallow does not mean "remove from Google" This is the costliest misunderstanding in the file. Disallow means "do not crawl". It says nothing about indexing. If another site links to a URL you have blocked, Google can still show that URL in results, usually with no description, because it was never allowed to read the page. Google's own documentation is blunt about this: robots.txt is not a mechanism for keeping a page out of search. There is a nastier side effect. Blocking a page also stops Google from seeing a noindex tag on that page, so the two instructions cancel each other out. If you want a page gone from results, let crawlers fetch it and serve a noindex robots meta tag, or return a proper 404 or 410 if the page is genuinely retired. And noindex written inside robots.txt has never done anything on Google's side since September 2019, when they confirmed they no longer support that rule, along with nofollow and crawl-delay. ## The errors we find on live sites The staging leftover is number one. `Disallow: /` gets copied from a development server to production and nobody notices for weeks. In WordPress, the same damage comes from the "Discourage search engines from indexing this site" checkbox under Settings, Reading. Make unticking it the first item on your launch checklist, not the last. Blocking assets is a close second. A blanket disallow on a theme or plugin directory stops Google fetching the CSS and JavaScript it needs to render the page, so it evaluates a broken layout. Let crawlers have your stylesheets, scripts and images. Then there is blocking a whole section to deal with duplicate URLs, usually filter or tracking parameters. It backfires, because a blocked page can never be read, which means the canonical tag inside it can never be read either. Canonical tags are the right tool for duplicates. Finally, watch the file's HTTP response. A missing robots.txt returning 404 is fine and simply means "crawl everything". A robots.txt returning a 500 error is not fine: Google may pause crawling the site until it can read the file again, which is one of the reasons we care so much about how a server behaves under load on our [shared hosting](/hosting). Worth knowing on WordPress too: if a real robots.txt file exists on disk, it wins, and whatever your SEO plugin shows you in its settings screen is ignored. ## What most small sites should be running Most sites need very little. Something like this is enough: ``` User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap_index.xml ``` The Sitemap line is the part with real value, since it points crawlers straight at your list of URLs. If you have not sorted that out yet, our notes on [XML sitemaps for small sites](/blog/xml-sitemaps-small-sites) cover what belongs in one. ## Check it after every change Open yoursite.com/robots.txt in a browser after any redesign, host move, plugin change or staging push. Read it line by line. Then use the URL inspection tool in Search Console on a handful of important pages, including your homepage and two or three money pages, and confirm none of them come back as blocked. Our walkthrough of [the Search Console indexing report](/blog/search-console-indexing-report) explains where blocked URLs surface once Google has recrawled. If your pages have dropped out of search and you are not sure whether robots.txt, a redirect or a server response is behind it, send us the URLs through [support](/support) and we will look at what the server is actually returning to crawlers. ## Sources - [Robots.txt Introduction and Guide, Google Search Central](https://developers.google.com/search/docs/crawling-indexing/robots/intro) - [Block Search Indexing with noindex, Google Search Central](https://developers.google.com/search/docs/crawling-indexing/block-indexing) - [A note on unsupported rules in robots.txt, Google Search Central Blog](https://developers.google.com/search/blog/2019/07/a-note-on-unsupported-rules-in-robotstxt)