robots.txt Mistakes That De-Index Whole Sites

A robots.txt file controls crawling, not indexing — and Google says so in its own words: "it is not a mechanism for keeping a web page out of Google." Nearly every expensive robots.txt mistake comes from believing otherwise.

The file is two lines of text that can remove a site from search results, and the single most common way that happens is not a subtle syntax error. It is Disallow: / surviving a launch.

---

What robots.txt actually does

It tells compliant crawlers which URLs they may request. That is the whole function, and Google's stated purpose for it is avoiding server overload rather than controlling what appears in results.

Two consequences follow, and both surprise people.

A disallowed page can still be indexed. Google's documentation is explicit: "A page that's disallowed in robots.txt can still be indexed if linked to from other sites." The URL and the anchor text used to link to it can appear in results — without a description, because Google was never permitted to read the page.

A disallowed page cannot be de-indexed by robots.txt. If a page is already in the index, blocking it in robots.txt does not remove it. It just prevents Google from re-reading it, which can freeze the existing entry in place.

Google's own recommendation for keeping a page out of results: a noindex meta tag or response header, password protection, or removing the page.

---

The mistake that cancels itself out

Disallowing a page in robots.txt and putting noindex on it.

This looks like belt and braces. It is the opposite.

noindex lives in the page — in a meta tag or an HTTP header. Google has to fetch the page to see it. If robots.txt forbids fetching, the directive is never read, and the page stays in the index indefinitely, usually as a bare URL with no description.

The correct sequence, when a page is already indexed and you want it gone:

1. Allow crawling — remove the robots.txt disallow
2. Add noindex to the page
3. Wait for Google to re-crawl and drop it
4. Only then, if you also want to save crawl budget, add the disallow back

Doing steps one and two in the wrong order is why pages people have "blocked" keep appearing for years.

---

The mistakes that cost the most

1. Disallow: / from staging

User-agent: *
Disallow: /

Two lines, entire site blocked. It gets written on a staging environment for good reason and copied to production at launch.

This is the single most expensive mistake in technical SEO, and it is invisible unless you look — the site works perfectly for humans while search traffic decays over weeks.

On WordPress there is a second route to the same outcome: Settings → Reading → "Discourage search engines from indexing this site". It is a checkbox, it survives launches routinely, and it produces the same result.

2. Blocking resources the page needs to render

Disallow: /wp-content/
Disallow: /assets/
Disallow: /*.js$

Google renders pages. Block the CSS and JavaScript and it renders a broken page — which it then judges on what it could see. This was once standard advice and is now actively harmful.

3. Blocking a URL you also want de-indexed

Covered above. The most common self-defeating configuration in the file.

4. Trying to use noindex inside robots.txt

User-agent: *
Noindex: /private/

Google does not support this. It is silently ignored, and the page you believed was excluded is not.

5. Blocking the sitemap or the sitemap's URLs

Blocking URLs listed in your own sitemap sends contradictory instructions — here are my pages, do not look at them. It also generates avoidable Search Console warnings.

6. A blanket rule that catches search crawlers

User-agent: *
Disallow: /

…written to stop AI training, and it stops Googlebot too. The correct way to separate AI crawlers by purpose is in llms.txt and AI crawlers.

7. Case and syntax errors

Paths are case-sensitive. Disallow: /Private/ does not block /private/. A missing leading slash, a stray space, or a rule under the wrong User-agent block fails silently — robots.txt does not report errors, it just does not do what you meant.

---

How the rules are actually applied

Four behaviours worth knowing, because they cause "but I wrote the rule" confusion.

Only one User-agent group applies. Google follows the most specific matching group and ignores the others. If you have a Googlebot block and a * block, Googlebot obeys only the first — including rules you assumed it inherited from *.

The most specific rule wins, not the last one. A longer, more specific path takes precedence regardless of order.

Allow can override Disallow. This is how you block a directory while permitting one file inside it:

User-agent: *
Disallow: /private/
Allow: /private/public-page.html

It must be at the root. example.com/robots.txt — a file at example.com/subfolder/robots.txt is ignored entirely. Subdomains need their own.

---

What to use instead

Goal	Correct method
Keep a page out of search results	noindex meta tag or response header, and allow crawling
Protect genuinely private content	Password protection or authentication — not robots.txt, which is a public file
Remove a page urgently	Search Console Removals tool for a temporary block, plus noindex for the permanent fix
Reduce crawl load	Disallow — this is what it is actually for
Handle duplicate URLs	Canonical tags, not blocking

Remember that robots.txt is public. Anyone can read yourdomain.com/robots.txt. Listing /admin/ or /private-client-files/ in it publishes a directory of the things you did not want found.

---

Checking yours

Read it first, with your own eyes. Open yourdomain.com/robots.txt. Most problems are visible immediately to anyone who reads the whole file, and this takes thirty seconds.

Then:

• Search Console → Settings → robots.txt shows the fetched version, when it was last read, and any parse problems
• URL Inspection on a specific URL tells you whether robots.txt is blocking it
• The Page Indexing report surfaces "Blocked by robots.txt" and "Indexed, though blocked by robots.txt" — the second is the warning that proves the file is not an indexing control. Decoded in why isn't my page indexed

Check it after every migration, host change, theme change and staging refresh. Those are the four events that overwrite it, and none of them announces that it has.

---

Frequently asked questions

Does robots.txt stop a page being indexed?
No. Google states plainly that robots.txt "is not a mechanism for keeping a web page out of Google," and that a disallowed page can still be indexed if other sites link to it — appearing without a description because Google was never allowed to read it. Use a noindex meta tag or header instead.

Why is my page indexed even though it is blocked in robots.txt?
Because blocking crawling is not the same as blocking indexing. Google respects the disallow and does not fetch the page, but it can still index the URL based on links pointing at it. Search Console reports this as "Indexed, though blocked by robots.txt".

Can I use noindex in robots.txt?
No. Google does not support a noindex directive in robots.txt; it is silently ignored. The noindex directive belongs in the page's meta tag or HTTP response header, and Google must be allowed to crawl the page in order to see it.

Why does my noindex not work?
Most often because the page is also disallowed in robots.txt. Google cannot read a directive inside a page it is not permitted to fetch, so the noindex is never seen. Allow crawling, keep the noindex, wait for a re-crawl, and only then consider blocking.

What is the most damaging robots.txt mistake?
Disallow: / copied from staging to production. It blocks the entire site, the site continues working normally for human visitors, and search traffic decays over weeks. On WordPress the "Discourage search engines" setting produces the same result.

Should I block CSS and JavaScript in robots.txt?
No. Google renders pages, and blocking the resources needed to render means it judges a broken version of your page. This was once common advice and is now actively harmful.

Is robots.txt case-sensitive?
Yes, for paths. Disallow: /Private/ does not block /private/. The file also fails silently — there is no error report when a rule does not match what you intended.

Where does robots.txt have to live?
At the root of the domain, as example.com/robots.txt. A file placed in a subfolder is ignored entirely, and each subdomain needs its own.

---

What to do next

Open yourdomain.com/robots.txt in a browser and read every line. Not a tool — read it. This is the single highest-yield thirty seconds available in technical SEO.

If you are on WordPress, also check Settings → Reading for the "Discourage search engines" box, because it produces the same outcome and is not in the file at all.

---

Related guides

• Technical SEO: what actually breaks rankings — where this sits in the four layers
• Why isn't my page indexed? A diagnostic — decoding the resulting statuses
• llms.txt and AI crawlers: block them or not? — separating AI bots correctly
• Search Console: the 5 reports that matter — where to verify all of this
• Schema markup for blogs: what to add and how — the other file people get wrong