Coder Legion - Recent posts tagged indexing

noindex vs robots.txt: Which to Use and the Mistake to Avoid

Fri, 23 Oct 2026 04:00:51 +0000

noindex is for keeping a page out of results; a robots.txt Disallow is for stopping crawlers from requesting URLs. The two aren't interchangeable. A disallowed URL can still appear in Google without a description, and a noindex tag on a disallowed page is never seen, so using both keeps pages in Google. That combination is the most common indexing mistake I see on small-business sites.

The guide explains the difference between discovery, crawling and indexing, and what each control does at which step, based on Google's documentation and the RFC 9309 standard. If you want something private, neither is right; it belongs behind a login or HTTP authentication.

I walk through fixing the double-blocked page (remove the Disallow, keep the noindex, let it be recrawled), the difference between the meta robots tag and the X-Robots-Tag header for PDFs and other files, and two things that don't work long term: a Noindex line in robots.txt (ignored since September 2019) and the Removals tool (temporary).

For AI, robots.txt is the main documented control for AI crawlers, while Google's own AI features follow the same noindex and snippet rules as Search, and Google-Extended controls Gemini use without affecting Search. I finish with common cases like staging sites and a decision table matching each goal to the right control.

Key takeaways

robots.txt controls crawling; noindex controls indexing
A noindex on a disallowed page is never seen
Use X-Robots-Tag for PDFs and non-HTML files
Protect staging sites with authentication, not robots.txt
Noindex in robots.txt has been ignored since 2019

Related on BuiltToWinWeb: canonical tags explained

Read the full guide on BuiltToWinWeb: Noindex vs robots.txt, and when to use each

A summary by Jacob Campbell of an article first published on built2winweb.com on October 7, 2026.

Canonical Tags Explained: Fixing Duplicate URLs for Google and AI

Sun, 18 Oct 2026 07:09:38 +0000

A canonical tag tells search engines which address is the primary one when identical content lives at several URLs. Google handles it as a hint rather than a rule, and duplicates are rarely deliberate: protocol and www variants, trailing slashes, tracking and filter parameters, print views and now Markdown versions of pages for AI agents. This guide explains how canonicals really work.

I start with why it matters: choosing which URL appears in results, consolidating link signals, keeping reporting clean and saving crawl time. Duplicate content isn't a penalty; the cost is dilution and loss of control. Then the four ways to declare a canonical, from strongest to weakest: a permanent redirect, rel=canonical in the head or an HTTP Link header (useful for PDFs and Markdown twins), and sitemap inclusion.

Google weighs several signals and may override you, so I explain how to check the result in URL Inspection by comparing the user-declared and Google-selected canonicals. There's a list of where duplicates come from with the right fix for each, noting that many are better solved with a redirect, and that paginated pages should keep their own canonicals.

The mistakes section covers canonicals pointing at noindexed, redirected or missing pages, everything canonicalised to the homepage, tags pushed into the body, conflicting canonicals and accidental cross-domain ones. I also cover multilingual sites and a closing checklist.

Key takeaways

A canonical is a strong hint, not a command
Redirect duplicates that don't need to exist
Use a Link header for PDFs and Markdown versions
Compare declared and Google-selected canonicals in URL Inspection
Never point a canonical at a noindexed, redirected or missing page

Related on BuiltToWinWeb: noindex vs robots.txt

Read the full guide on BuiltToWinWeb: Canonical tags explained

A summary by Jacob Campbell of an article first published on built2winweb.com on October 7, 2026.

Crawled - Currently Not Indexed: What to Fix | FloxoLab

Tue, 06 Oct 2026 15:50:00 +0000

Google still defines Crawled - currently not indexed in the Page indexing report as a page that Google crawled but did not index. Google adds that the page may or may not be indexed later and that there is no need to resubmit the URL for crawling. The label confirms a crawl happened. It does not identify one universal defect, and it does not promise that a technically indexable page will be selected for the index.

The useful question is not "How do I clear this warning?" It is "What should happen to this specific URL?" An important service page, an expired campaign page, a duplicate product filter, and a thin archive should not receive the same treatment. Start with the intended outcome.

Choose the right outcome before changing the page

What you find	Best next action	Why
Unique, useful page with a technical or rendering conflict	Fix	Keep the URL and remove the evidence-backed obstacle
Page overlaps a stronger URL that serves the same intent	Consolidate	Merge useful material and use a permanent redirect when the old URL is retired
Useful to visitors but should not appear in search	Noindex	Keep it accessible while explicitly excluding it from search results
Temporary or recent URL with no conflicting signals	Leave it alone and monitor	Crawling and indexing are separate, and inclusion is not guaranteed on a fixed schedule
Obsolete page with no relevant replacement	Remove	Return a real 404 or 410 instead of preserving an empty 200 page

Do not bulk-request indexing first. Google says repeated requests for the same URL do not make crawling faster, and a crawl request does not guarantee inclusion. Use the request only after a meaningful fix or update.

Step 1: confirm the exact URL and current state

In Search Console, open Indexing → Pages → Crawled - currently not indexed, export the examples, and pick a small representative sample. Separate important pages from parameters, feeds, media URLs, old campaign paths, and other URLs that were never meant to rank. A large count can hide several unrelated patterns.

Paste one complete URL into URL Inspection. The indexed result reports what Google knows from its stored data, not necessarily what is live now. Record the last crawl time, crawl result, whether indexing was allowed, the user-declared canonical, and any referring page or sitemap information shown. Then run Test live URL if the page has changed or you suspect the stored crawl is stale.

The live test answers a narrower question: can Google fetch and process the current page? Google explicitly says it cannot predict indexing success, canonical selection, or the Crawled - currently not indexed state. A green "URL is available to Google" result is useful, but it is not proof that the URL will be indexed.

Step 2: check indexability and the response Google receives

For a page you want indexed, the live response should be a stable 200, accessible without a login, allowed by robots.txt, and free of a noindex rule in both the HTML and HTTP headers. Check the rendered HTML that URL Inspection received, not only the source your browser shows.

HTTP status: the intended page returns 200, not a redirect loop, intermittent 5xx, 403, or authentication response.
Robots access: Googlebot can crawl the page and the CSS, JavaScript, and other resources needed to render its main content.
Indexing directive: there is no accidental robots meta tag or X-Robots-Tag: noindex.
Rendered content: the screenshot and rendered HTML contain the main heading, body, links, and useful page-specific information.
Error content: the page is not a product-missing message, empty search result, or not-found template returning 200.

Robots.txt is not a reliable way to remove a web page from search. Google may still know and show a blocked URL, and a blocked crawler cannot see a noindex rule on the page. If the URL must stay public but out of search, allow crawling and use noindex. If it is private, require authentication.

Step 3: align canonical, sitemap, and redirect signals

Canonicalization happens during indexing, so the live test cannot tell you which canonical Google will choose. You can still remove obvious conflicts. A page meant to stand alone should normally have a self-referencing canonical. Its internal links, sitemap entry, redirects, and alternate versions should consistently point to the same preferred clean URL.

Check for common contradictions: the page declares another URL as canonical, the sitemap lists a parameter or HTTP version, internal links point through a redirect, or several near-identical URLs all declare themselves canonical. Google treats canonical annotations, redirects, sitemap inclusion, and other signals as inputs, not commands. Conflicting signals make the preferred version less clear.

A sitemap helps Google discover preferred URLs. It does not force Google to crawl or index them. Keep only canonical, indexable URLs in the sitemap, use accurate lastmod values when content actually changes, and do not resubmit the file as a substitute for fixing the page.

If the submitted file lists redirect sources or URLs that canonicalize elsewhere, use the sitemap redirect and canonical cleanup workflow to audit the final response, replace each obsolete entry, and repair the generator that keeps adding it.

Step 4: make sure the page is discoverable in context

Every important page should have at least one crawlable link from another page, and a business-critical page usually deserves several relevant paths. Use normal <a href="https://floxolab.com/seo/guides/..."> links with descriptive anchor text. Do not rely only on a sitemap, a JavaScript click handler, an orphaned card loaded after interaction, or a navigation search box.

Link from pages that naturally explain the relationship. A service page can link to the detailed guide that supports it. A related guide can point to the next troubleshooting step. This helps users navigate and gives Google clearer context about why the URL exists. For the broader site-level check, use the Search Console pre-audit to separate isolated URLs from a wider indexing pattern.

Step 5: decide whether the URL deserves its own result

Technical cleanliness is necessary, but it is not a guarantee of selection. Compare the page with the strongest page on your own site that answers the same search need. If both URLs would satisfy the same person with substantially the same answer, creating a second canonical page may not help anyone. When both URLs already earn impressions, use the keyword cannibalization workflow in Search Console to compare their query, intent, and performance before choosing a keeper.

Ask concrete questions instead of counting words:

Does the page solve a distinct user problem that no stronger page already solves?
Does it contain original evidence, examples, decisions, or first-hand detail beyond a generic summary?
Is the main answer complete without requiring the visitor to click through to another page?
Are the title and H1 accurate descriptions of the content rather than keyword variants of another page?
Would a customer still find the page useful if search traffic never arrived?

Google recommends people-first content with original information, substantial coverage, clear sourcing, and a satisfying answer. Adding 500 generic words does not prove any of those things. Improve the page only when you can add missing value, such as a tested process, real comparison, local constraint, original example, calculation, or evidence-backed decision framework.

Step 6: improve, merge, noindex, or remove

Improve the existing URL

Keep the URL when its intent is distinct and the page matters. Fix the specific technical problem, replace placeholder or duplicated sections, add the evidence the reader needs, strengthen relevant internal links, and update the title or heading only when the page's actual focus changed. Preserve the clean URL unless there is a compelling reason to move it.

Merge and permanently redirect

If two pages serve the same intent, move any genuinely useful material into the stronger page and use a server-side 301 or 308 redirect from the retired URL to the relevant destination. Do not redirect every removed page to the homepage. The target should be a sensible replacement for the visitor.

If indexing is only one part of a broader content cleanup, the old page decision framework adds search demand, business value, backlinks, and replacement intent to the choice.

Keep it for users but noindex it

Some pages are useful without belonging in search: internal search results, thank-you pages, account steps, filtered views, or short-lived campaign states. Use a robots meta tag or X-Robots-Tag with noindex, leave the URL crawlable so Google can see the rule, and remove it from the sitemap.

Remove it correctly

If the page is obsolete and has no replacement, return 404 or 410. A friendly not-found design can still help visitors, but the HTTP response must tell crawlers the resource is gone. Avoid an empty or error-like template that returns 200, which Google may classify as a soft 404.

Step 7: request indexing only after a meaningful change

After fixing an important individual URL, run the live test again. Confirm that the current response, rendered content, directives, and canonical declaration match the intended state. Then use Request indexing once. For many changed URLs, update the sitemap and let normal crawling work instead of submitting them one by one.

Google documents that crawling can take from a few days to a few weeks. It also states that a request does not guarantee immediate crawling or inclusion at all. There is no defensible fixed indexing SLA. Record the request date, the change made, and the next review date so you can distinguish waiting from repeated, evidence-free resubmission.

How to validate the result without chasing the report

Check	What it can tell you	What it cannot prove
URL Inspection indexed result	Google's stored index status, last crawl, and selected signals	That the live page is unchanged or currently ranking
Test live URL	Whether the current page can be fetched and appears technically indexable	Future indexing or canonical selection
Page indexing report	Patterns and counts across known URLs	A live verdict for every example URL
Search for the exact URL	Whether that URL currently appears for the test	Stable rankings, traffic, or impressions
Performance report	Impressions and clicks after Google begins serving the page	Guaranteed visibility for a newly indexed URL

Review the exact URL in URL Inspection and note the Page indexing report's own last-updated date. Do not keep changing a good page every few days because a grouped report has not moved. If the URL remains excluded after a fresh crawl, compare the rendered version and canonical signals again, then revisit whether the page is distinct enough to keep.

Practical questions

Is Crawled - currently not indexed a penalty?

The status itself is a page indexing classification, not a manual action notice. Check Search Console's separate Manual actions and Security issues reports if you suspect a site-level enforcement or security problem.

Will adding more words fix it?

Not by itself. Add information only when it makes the answer more original, complete, or useful. A longer duplicate remains a duplicate, and generic filler does not resolve a canonical, rendering, noindex, or soft 404 problem.

Should every URL in the report be indexed?

No. Redirects, obsolete URLs, feeds, filtered combinations, account steps, and intentionally excluded pages may be correct non-indexed outcomes. Prioritize URLs that are important to users and have a clear, unique search purpose.

Sources checked

Product behavior and documentation were verified against official Google sources on July 21, 2026.

<p class="cta

Indexerror: index 0 is out of bounds for axis 0 with size 0 /Needs improvement

Mon, 10 Aug 2026 06:23:51 +0000

The cause for Indexerror: index 0 is out of bounds for axis 0 with size 0 is when trying to access the first element stored inside the first dimension array. However, such a numpy array is empty and hence inaccessible. This tutorial shows an example of how the error arise and later, the ways of handling such an error such as first checking if the numpy array is empty or not before access, or using try/except.

Example of the Error

Let's take an example of a numpy array that we want to access the first element from.

import numpy as np
exArray=np.array([])
element1=exArray[0]

When we run the above code, we get the following output:

line 4, in 
element1=exArray[0]
IndexError: index 0 is out of bounds for axis 0 with size 0

As we see, the error "IndexError: index 0 is out of bounds for axis 0 with size 0" occurs when trying to access the first element stored inside the first dimension array. However, such a numpy array is empty and hence inaccessible.

Note The numpy array "exArray" was created with an empty list, which resulted in the error. It is crucial to check the initialization of the array and ensure that it is not empty before attempting to access its elements.

Solution to the Error

There are multiple ways to solve this error, let's take a look at some of the most effective solutions.

Solution One: Checking if the array is empty or not before accessing

One of the most straightforward solutions is to check the size of the array before trying to access it. If the size of the array is 0, then the array is empty and we should not try to access it.

import numpy as np
exArray=np.array([])
if exArray.size>0:
print(exArray[0])
else:
print('The size of the array is 0')

The output would be:

The size of the array is 0

As we see, we are able to check if the size of the array is 0 or not before trying to access it. In this way, we can avoid the error from occurring.

Solution Two: Change the shape of the array

Another solution is to change the shape of the array . By giving the array a shape, we are able to access it without any error.

import numpy as np
exArray=np.zeros((0,4),dtype=int)
print(exArray.shape)

The output would be:

(0, 4)

As we see, by giving the array a shape, we are able to access it without any error.

Solution Three: Use Try/Except

Using try/except block to catch the error is another effective solution. In this way, if the error occurs, we can handle it by providing a message or any other action that we want to take.

import numpy as np
exArray=np.array([])
try:
print(exArray[0])
except IndexError:
print('The index 0 is empty')

The output would be:

The index 0 is empty

As we see, by using the try/except block, we are able to catch the error and handle it.

Tip A good practice to avoid this error is to always initialize the array with a default size or shape, and check the array's size before trying to access its elements. This way, you can ensure that the array is not empty and avoid the IndexError from occurring. Additionally, it is always a good idea to use try/except blocks to catch any unexpected errors, and handle them accordingly.

Conclusion

Hopefully you have now learned more concerning the Indexerror: index 0 is out of bounds for axis 0 with size 0 error that occurs when trying to access an element from an empty numpy array. We have also learned about some effective solutions such as checking if the array is empty before accessing it, changing the shape of the array, and using a try/except block to catch the error. Remember, it is important to check if the array is empty before trying to access any element from it to avoid the error.

References

*Numpy documentation on numpy.array - https://numpy.org/doc/stable/reference/generated/numpy.array.html

*Numpy documentation on numpy.size - https://numpy.org/doc/stable/reference/generated/numpy.ndarray.size.html

*Python documentation on try and except - https://docs.python.org/3/tutorial/errors.html

Index was outside the bounds of the array

Thu, 21 May 2026 12:28:19 +0000

Today, we're going to fix a C# Error: Index was outside the bounds of the array that usually occurs when we try to access the index less than 0 or greater than or equal to the array length - 1. In this study blog, we'll explore why it happens and how to resolve it.

What is "Index was outside the bounds of the array" #

This error happens when we try to access the elements in the array using invalid index. Let's see an example:


int[] numbers = new int[] { 1,2,3,4,5,6,7,8,9 };
for(int i = 0; i <= numbers.Length; i++)
{
    Console.WriteLine(numbers[i]);
}
  </span>

This is the output:


1
2
3
4
5
6
7
8
9
System.IndexOutOfRangeException: 'Index was outside the bounds of the array.'

Why? Because the numbers array has 9 elements, and thus its length is 9. However, in programming, each collection like array has indices in the range of [0 - N-1], where N = array.length and we can access each element in the array within that range. In the above example, after accessing index 8 with element 9 in numbers array, we tend to access element at index 9, which is an invalid index for numbers array. That's why this error occur.

Let's look at another example:


int[] numbers = new int[] { 1,2,3,4,5,6,7,8,9 };
numbers[-1] = 10;

This is the output:


System.IndexOutOfRangeException: 'Index was outside the bounds of the array.'

Why? we're trying to assign value to the element of numbers array at index -1, which is also an invalid index for numbers array. Hence, the error will also occur.

Causes #

The error, Index was outside the bounds of the array is actually under IndexOutOfRangeException Class in C#. IndexOutOfRangeException: The exception is thrown when the attempt to access the element of a collection or an array using the index outside its bounds is made. Hence, System.IndexOutOfRangeException: 'Index was outside the bounds of the array.' occurs when we're trying to access the element of an array using invalid index. Don't fret; it can be simply resolved.

Note Every collection such as list, array or the other has only indices in the range of [0, N-1], where N is the length of the array itself.

Solution #

The solution is pretty simple, we just need to ensure we're accessing the elements of the array using the indices within its bounds. Let's say:


int[] numbers = new int[] { 1,2,3,4,5,6,7,8,9 };
for(int i = 0; i < numbers.Length; i++)
{
    Console.WriteLine(numbers[i]);
}

This is the output:

or you may also do like this:


int[] numbers = new int[] { 1,2,3,4,5,6,7,8,9 };
for(int i = 0; i <= numbers.Length-1; i++)
{
    Console.WriteLine(numbers[i]);
}
  </span>

This is the output:

Both examples above are having the same output. What we do is to ensure that the iteration will only be in the range of [0 - N-1], where N = array.length. The numbers array has the length of 9, and thus can only iterate from 0 to 8. These are the valid indices for the numbers array.

Tip You may also use GetLowerBound(Int32) and GetUpperBound(Int32) to avoid this error.


int[] numbers = new int[] { 1,2,3,4,5,6,7,8,9 };
for(int i = numbers.GetLowerBound(0); i <= numbers.GetUpperBound(0); i++)
{
    Console.WriteLine(numbers[i]);
}
  </span>

This is the output:

For the case above, GetLowerBound(0) is used to get the index of the first element, which is 0 while GetUpperBound(0) is used to get the index of last element. Hence, we can omit the error by using these two functions together.

Note The dimension for both GetLowerBound(int dimension) and GetUpperBound(int dimension) is determined by the array type, whether it is one-dimensional array or two-dimensional array as they have the different Rank. Make sure the dimension has the value of greater than 0 or Rank - 1.

The Conclusion #

In this study blog, we've explore what is Index was outside the bounds of the array in C#, why it happens and how to resolve it. You should always be aware with the indices of the array when you want to access its element. Make sure not to access index outside of its bound, which is the range [0 - N - 1], where N = length of the array. Thanks!

The Reference #

For further information, please visit:

IndexOutOfRangeException: https://learn.microsoft.com/en-us/dotnet/api/system.indexoutofrangeexception?view=net-7.0
GetLowerBound(Int32) in LINQ: https://learn.microsoft.com/en-us/dotnet/api/system.array.getlowerbound?view=net-7.0#system-array-getlowerbound(system-int32)
GetUpperBound(Int32): https://learn.microsoft.com/en-us/dotnet/api/system.array.getupperbound?view=net-7.0#system-array-getupperbound(system-int32)

Coder Legion - Recent posts tagged indexing

noindex vs robots.txt: Which to Use and the Mistake to Avoid

Key takeaways

Canonical Tags Explained: Fixing Duplicate URLs for Google and AI

Key takeaways

Crawled - Currently Not Indexed: What to Fix | FloxoLab

Choose the right outcome before changing the page

Step 1: confirm the exact URL and current state

Step 2: check indexability and the response Google receives

Step 3: align canonical, sitemap, and redirect signals

Step 4: make sure the page is discoverable in context

Step 5: decide whether the URL deserves its own result

Step 6: improve, merge, noindex, or remove

Improve the existing URL

Merge and permanently redirect

Keep it for users but noindex it

Remove it correctly

Step 7: request indexing only after a meaningful change

How to validate the result without chasing the report

Practical questions

Is Crawled - currently not indexed a penalty?

Will adding more words fix it?

Should every URL in the report be indexed?

Sources checked

Indexerror: index 0 is out of bounds for axis 0 with size 0 /Needs improvement

Table of Contents:

Example of the Error

Solution to the Error

Solution One: Checking if the array is empty or not before accessing

Solution Two: Change the shape of the array

Solution Three: Use Try/Except

Conclusion

References

Index was outside the bounds of the array

Table of Contents:

What is "Index was outside the bounds of the array" #

Causes #

Solution #

The Conclusion #

The Reference #