Coder Legion - Recent posts tagged robots-txt

noindex vs robots.txt: Which to Use and the Mistake to Avoid

Fri, 23 Oct 2026 04:00:51 +0000

noindex is for keeping a page out of results; a robots.txt Disallow is for stopping crawlers from requesting URLs. The two aren't interchangeable. A disallowed URL can still appear in Google without a description, and a noindex tag on a disallowed page is never seen, so using both keeps pages in Google. That combination is the most common indexing mistake I see on small-business sites.

The guide explains the difference between discovery, crawling and indexing, and what each control does at which step, based on Google's documentation and the RFC 9309 standard. If you want something private, neither is right; it belongs behind a login or HTTP authentication.

I walk through fixing the double-blocked page (remove the Disallow, keep the noindex, let it be recrawled), the difference between the meta robots tag and the X-Robots-Tag header for PDFs and other files, and two things that don't work long term: a Noindex line in robots.txt (ignored since September 2019) and the Removals tool (temporary).

For AI, robots.txt is the main documented control for AI crawlers, while Google's own AI features follow the same noindex and snippet rules as Search, and Google-Extended controls Gemini use without affecting Search. I finish with common cases like staging sites and a decision table matching each goal to the right control.

Key takeaways

robots.txt controls crawling; noindex controls indexing
A noindex on a disallowed page is never seen
Use X-Robots-Tag for PDFs and non-HTML files
Protect staging sites with authentication, not robots.txt
Noindex in robots.txt has been ignored since 2019

Related on BuiltToWinWeb: canonical tags explained

Read the full guide on BuiltToWinWeb: Noindex vs robots.txt, and when to use each

A summary by Jacob Campbell of an article first published on built2winweb.com on October 7, 2026.

Content Signals and RSL: Stating How AI May Use Your Content

Sun, 18 Oct 2026 07:09:29 +0000

robots.txt only decides whether a crawler may fetch a path. It's silent on what happens next: whether the content appears in a search result, gets quoted in an AI answer or is used to train a model. Two newer mechanisms address that, and this guide explains both and is straight about their limits.

Content Signals, introduced by Cloudflare in September 2025, adds a Content-Signal line to robots.txt with three settings: search, ai-input (using content in a real-time AI answer) and ai-train. Each is yes or no, and a missing key means no stated preference. The split matters because search and AI answers send visitors and citations, while training only helps you indirectly, and robots.txt alone can't express "answer questions with my content but don't train on it" when one crawler serves several purposes. On this site I publish yes to all three.

RSL (Really Simple Licensing), also announced in September 2025, defines a machine-readable licence stating what AI systems may do with content and on what terms, from free with attribution to pay per crawl. We publish one here as an experiment.

The limits are important. Neither is enforcement, neither reaches content already collected, and real enforcement lives in CDN bot management and verified-bot programmes. I finish with a short list of what to publish and how to keep it consistent with your user-agent rules.

Key takeaways

robots.txt controls fetching, not later use
Content Signals covers search, ai-input and ai-train
RSL publishes machine-readable licence terms
Both state preferences; neither enforces them
Make sure your CDN isn't overwriting your signals

Related on BuiltToWinWeb: Cloudflare's AI crawl controls

Read the full guide on BuiltToWinWeb: Content Signals and RSL explained

A summary by Jacob Campbell of an article first published on built2winweb.com on October 5, 2026.

Most sites that blocked AI blocked the wrong thing

Fri, 31 Jul 2026 19:40:00 +0000

Sometime in the last two years you probably added a rule to robots.txt to keep AI crawlers out. A lot of us did. The intent was reasonable: don't train on my content without asking.

Go look at what you actually wrote. There is a good chance it reads something like this:

User-agent: *
Disallow: /

Or it is a wildcard rule inherited from a config written for Google in 2019, which predates every agent this is supposed to be about.

Either way, the result is not what you intended.

OpenAI runs several agents and they do different jobs

GPTBot — crawls for model training
OAI-SearchBot — crawls to build the index behind ChatGPT search
ChatGPT-User — fetches a page when a user's prompt triggers it

They are separate on purpose. Blocking all of them opts you out of training and removes you from ChatGPT's search index. The first was the goal. The second was collateral, and it means your content can no longer be cited when someone asks a question your site answers.

If what you wanted was "don't train on me, but do cite me":

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

That is a defensible position. So is blocking everything. So is allowing everything. The problem is that most robots.txt files I look at are not expressing any of those positions — they are expressing a misunderstanding.

The part that surprises people

Fixing this does not put your products in ChatGPT's feed-driven shopping results, because that surface is not crawled.

Product data for it reaches ChatGPT through a feed the merchant pushes to OpenAI. In March 2026 OpenAI moved away from Instant Checkout and put its weight behind discovery instead — get the catalog data right, let the purchase complete on the merchant's own site.

But there is a third route that most write-ups miss, and it is the one that makes the robots.txt question matter more, not less. ChatGPT also runs Shopping Research, an agentic mode that browses in real time. OpenAI says its results are organic and based on publicly available retail sites — reading product pages directly and citing sources. Merchants who want to be eligible go through a separate allowlist.

So there are three pipelines, and people routinely collapse them into one:

crawlers → citations in conversational answers
feeds → placement in feed-driven shopping results
live browsing → inclusion in Shopping Research, which reads your actual pages

Blocking crawlers costs you the first and the third. Google is closer to the simple version: AI Mode draws on a Shopping Graph built mainly from Merchant Center feeds, with your site markup contributing rather than deciding.

Perplexity is the odd one out. PerplexityBot indexes your actual HTML, so page quality still translates into visibility there with no feed and no application. Worth knowing, though: Perplexity's own docs say Perplexity-User — the live fetch triggered by a prompt — generally does not honour robots.txt. Whether that is defensible is a separate argument, and Cloudflare and Perplexity have been having it publicly since August 2025.

Check your logs, not your config

grep -Eo "GPTBot|OAI-SearchBot|ChatGPT-User|PerplexityBot|Googlebot" \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn

This matters more than reading your own robots.txt, because firewall-level blocking is invisible there. Your rules can be perfectly permissive while Cloudflare or your host rejects the request before anything is served. The log is the only ground truth about what is actually reaching you.

Where I land, and where I might be wrong

I think the crawler distinction is worth getting right regardless of your position on AI training, because "I opted out of citations by accident" is nobody's actual position.

What I am less sure about: whether being cited by ChatGPT is worth much. Citation rarely converts to a click — the answer usually satisfies the user in place. There is a real argument that being indexed by an engine that answers on your behalf is a bad trade, and that the accidental block was the right call arrived at by the wrong route.

I do not think that argument is obviously wrong. I just think it should be a decision rather than a typo.

So: did you block AI crawlers deliberately, and did you check what you actually blocked?

I build open-source AEO modules for Magento, so I have a stake in people caring about this. Treat it as a claim to verify rather than accept — the log command above is the verification.

Longer version covering all three engines: angeo.dev