menu

Recent posts tagged robots-txt

Level UI • 1 day • in Articles • 9 min read
An MCP server is not a new API. It is a thin layer over the one you already have
Somewhere in the last year, "do we need an MCP server?" joined the list of questions that arrive in a brief before anyone has said what the thing is for. It is usually framed as MCP vs API, as though the Model Context Protocol were a newer kind of AP...
post-cover-33074
  • ai
  • api
  • webdev
  • security
9 Reactions 7 Comments
Built2Win • Oct 22 • in Articles • 1 min read
noindex vs robots.txt: Which to Use and the Mistake to Avoid
noindex is for keeping a page out of results; a robots.txt Disallow is for stopping crawlers from requesting URLs. The two aren't interchangeable. A disallowed URL can still appear in Google without a description, and a noindex tag on a disallowed pa...
  • robots-txt
  • noindex
  • indexing
  • technical-seo
2 Reactions 0 Comments
Built2Win • Oct 18 • in Articles • 1 min read
Content Signals and RSL: Stating How AI May Use Your Content
robots.txt only decides whether a crawler may fetch a path. It's silent on what happens next: whether the content appears in a search result, gets quoted in an AI answer or is used to train a model. Two newer mechanisms address that, and this guide e...
  • ai-crawlers
  • content-licensing
  • robots-txt
  • ai-search
0 Reactions 0 Comments
angeo • Jul 31 • in Articles • 3 min read
Most sites that blocked AI blocked the wrong thing
Sometime in the last two years you probably added a rule to robots.txt to keep AI crawlers out. A lot of us did. The intent was reasonable: don't train on my content without asking. Go look at what you actually wrote. There is a good chance it reads...
post-cover-23902
  • seo
  • ai
  • webdev
  • robots-txt
8 Reactions 8 Comments
Tom Smithverified • 23 hours • in Articles • 9 min read
Developer Weekly Briefing — October 30, 2026
Security dominated this week, but not in the usual way. Two active campaigns turned GitHub configuration files into attack surfaces. Researchers demonstrated that the built-in browsers inside AI coding tools can read your SSH keys. And at Oracle AI ...
post-cover-33104
  • agenticai
  • devsecops
  • github
  • localai
6 Reactions 3 Comments
To see more, click for the full list of posts or popular tags.
chevron_left
🔥 3 posts • 👩‍💻 2 active writers
✨ Want to write in this topic?
Share your learning. Get discovered. Build your profile.
Join CoderLegion
💡 Why developers use CoderLegion
Track learning • Share posts • Get ranked publicly
Create your profile