noindex vs robots.txt: Which to Use and the Mistake to Avoid

Fri, 23 Oct 2026 04:00:51 +0000

noindex is for keeping a page out of results; a robots.txt Disallow is for stopping crawlers from requesting URLs. The two aren't interchangeable. A disallowed URL can still appear in Google without a description, and a noindex tag on a disallowed page is never seen, so using both keeps pages in Google. That combination is the most common indexing mistake I see on small-business sites.

The guide explains the difference between discovery, crawling and indexing, and what each control does at which step, based on Google's documentation and the RFC 9309 standard. If you want something private, neither is right; it belongs behind a login or HTTP authentication.

I walk through fixing the double-blocked page (remove the Disallow, keep the noindex, let it be recrawled), the difference between the meta robots tag and the X-Robots-Tag header for PDFs and other files, and two things that don't work long term: a Noindex line in robots.txt (ignored since September 2019) and the Removals tool (temporary).

For AI, robots.txt is the main documented control for AI crawlers, while Google's own AI features follow the same noindex and snippet rules as Search, and Google-Extended controls Gemini use without affecting Search. I finish with common cases like staging sites and a decision table matching each goal to the right control.

Key takeaways

robots.txt controls crawling; noindex controls indexing
A noindex on a disallowed page is never seen
Use X-Robots-Tag for PDFs and non-HTML files
Protect staging sites with authentication, not robots.txt
Noindex in robots.txt has been ignored since 2019

Related on BuiltToWinWeb: canonical tags explained

Read the full guide on BuiltToWinWeb: Noindex vs robots.txt, and when to use each

A summary by Jacob Campbell of an article first published on built2winweb.com on October 7, 2026.

Coder Legion - Recent posts tagged noindex

noindex vs robots.txt: Which to Use and the Mistake to Avoid

Key takeaways