Coder Legion - Recent posts tagged search-engine

SEO Made Simple: Best Website Builders for Beginners

Thu, 29 Oct 2026 07:06:24 +0000

Want to create a website for your company but scared of the technological difficulties? You are not alone. Many newcomers are intimidated by the coding, hosting and website settings. The good news is that it is much easier to establish a website nowadays and you don’t need any technological expertise. Today’s website builders allow you to build pages, add graphics, display your services and publish content - without writing a line of code. But getting your website up is only half the fight. You need people to know about it, too.

And that’s why it really matters to pick the best website builder for SEO success. You want a platform that does more than give you gorgeous templates and easy editing tools. You want to develop a site that’s easy for visitors to navigate, and that search engines can comprehend.

Why SEO is Important for Your Website

Imagine that you build a business, and you have a lot of wonderful things, but no signboard. People could pass right past you without knowing what you have to give. There is the same difficulty for a website with no search exposure.

Search engine optimization (SEO) is a strategy to assist the search engines comprehend your website and match your content with searches. This can be a terrific approach for small businesses, freelancers and new entrepreneurs to reach consumers who are actively looking for their products or services.

Paid ads stop producing traffic as soon as a campaign ends. Valuable content can continue to drive traffic from organic search for a long period. It might not happen immediately away, but if you continue with it, you can enhance your visibility.

SEO is not about putting keywords on your sites. It offers straightforward navigation, good content, a mobile friendly design and fast loading pages. The correct website builder can make these essentials easier to handle, so you can spend more time focusing on your business.

What Is An SEO Friendly Website Creator?

Not all website creators provide you the same control. Some make it easy to construct a page but limit your ability to alter critical SEO parameters. Others are more versatile as your site grows.

Look for tools that allow you to alter page names and meta descriptions, make meaningful URLs, manage headings, and add alt text to photos. This makes your information easier to retrieve and helps search engines interpret it.

Mobile-responsiveness matters, too. Your site should work on phones, tablets and computers without visitors having to pinch and zoom or struggle with menus. Loading speed is also crucial because consumers don’t want to wait for a page to open.

It also has important features like as SSL security, sitemap generating, redirects and Google Search Console integration. You don’t have to have all the bells and whistles from the get-go. Choose a platform that gets the basics right and enables you improve your website without getting into complex intricacies.

Picking an Easy Website Builder Online

If you are making a website for the first time, you want to get started without having to spend weeks learning how everything works. A Easy website builder online should let you construct pages easily, add content and update information.

Drag and drop editing can be really handy. Place text, graphics and buttons visually without scripting. Get a good start with ready to use templates instead of a blank screen.

And don’t forget to plan ahead for what you’ll need. You may start with a home page and several service pages, then want to add a blog, new services or landing pages. Find a builder who can deal with these alterations.

Make sure to check pricing before signing up. Your plan might include hosting, custom domains, security and extra services or might charge for them separately. This way you'll know the full price up front and won't have any nasty surprises.

Ultimately the proper platform needs to be easy to use today, and versatile enough to grow with your organization.

Website Builders Comparison for Beginners

It is reliant upon your needs. Each platform has its own use cases. So it’s important considering what each platform has to offer before you chose.

Wix is popular with beginners because of its visual editor, templates and built-in SEO capabilities. This is a fantastic solution for small enterprises, personal websites and service providers that desire a relatively simple editing experience.

WordPress is highly adaptable, especially for blogs and content-heavy webpages. There are many themes and plugins so there is a lot of potential for customization. However, hosting, updates and security can be additional work depending on your setup.

Squarespace is known for its sleek templates and easy design experience. It might be fantastic for creative professionals, consultants and enterprises who want to have a good looking internet presence without spending a lot of time on design.

Above all, Shopify is an ecommerce platform. It combines the functionality of an online store and necessary SEO settings, helpful for selling things, managing orders and receiving payments. But it’s eCommerce features could be overkill for a simple service site.

There is no one correct response for each person. Don't pick a platform just because it's popular—consider your budget, website goals, and how much control you need.

How to Create a Search Engine Friendly Website

When you choose a builder, you receive the tools, but how you use them is what makes the difference.

Each page needs to have a purpose. Your site should tell customers what your firm is, service pages should provide the essential material, and blog posts should answer questions that your audience regularly ask.

AEO & AIO: Getting ready for Today Search

More people are using AI technologies to ask inquiries and source information beyond traditional search engines. So it’s worth looking into Answer Engine Optimization (AEO) and Artificial Intelligence Optimization (AIO) in addition to SEO.

The AEO is all about answering particular questions in a clear way. AIO is all about making content useful, relevant, and intelligible to AI-powered systems. Neither one guaranties that your website will be featured in an AI-generated answer, but both encourage better content.

Answer Questions Directly, Use Descriptive Headlines, and Give Simple Explanations

For example, if you are describing website costs, first answer the big question and then get into the details. Accuracy matters too.

Keep your material fresh. Use structured data if possible and your platform allows it. Most importantly, create useful content, not just pages full of keywords. Your visitors, regardless of whether they come to your website via search or otherwise, require clear, trustworthy content.

Top Mistakes to Avoid When Building Your Website

It’s easy to get caught up in fancy animations, enormous graphics and fancy layouts. But too many design features can slow a website down and make it harder to explore. Usually, a tidy layout and understandable menu works best.

Another mistake is to build a website with a lot of pages to make it bigger. Each page should have a purpose, whether that’s explaining a service, answering a question or moving a customer ahead.

And don't forget to add the basic SEO settings. Avoid problems quickly with missing site names, broken links and confused url’s. Take a look at these before you start, and then refer back to them as your site grows.

and last of all, be patient. SEO is a marathon. The results you get will be depending on competition, quality of your material and other variables. Don’t anticipate instant rankings, expect incremental development.

Monitor SEO Progress Over Time

Once your site is live, tools such as Google Search Console will start to tell you what people are searching for to get to your pages and any potential technical difficulties. Analytics devices inform you how people engage with your website.

Don't just measure visits. For a business, questions, reservations, form submissions and sales can be better measures of success. Be sure to check your content often, update information that is no longer relevant and improve pages according to the demands of your visitors.

You don’t have to do it all at one time. Over time, little, incremental enhancements can make your website more valuable and effective.

ItWeb: Web Site Building Was Never Easier

Your firm doesn’t need to wrestle with sophisticated coding to get online. Imbibe Technologies runs ItWeb, a website development platform which features an inbuilt page builder and a no-code approach. A great location for businesses and individuals to start when looking for a straightforward way to develop and maintain their web presence.

Build Smart, Get Found, Grow Online

Your initial site doesn’t need to be perfect. It should explain visitors who you are, what you do and how to go about it.

Pick a website builder that you like, has the SEO tools that you need and has capacity for you to expand. Concentrate on valuable content, easy navigation and a pleasant experience from any device. I have to track my progress and make changes as I figure out what works.

Want to design your website without writing a single line of code? Discover ItWeb and start your adventure to build, manage and develop your online presence.

Keep Search Clean: Block Search Engine Spam with uBlacklist and Community Rule Sets

Sun, 20 Sep 2026 14:31:02 +0000

If you use Google, DuckDuckGo, or Bing regularly, you have likely run into programmatic search spam: auto-generated "doorway" pages, scraper blogs stealing content, DGA (Domain Generation Algorithm) random domain networks, and affiliate redirects. They waste time, clutter search results, and occasionally host malicious scripts or phishing landers.

While reporting these domains to registrars and search engines helps clean up the web for everyone, you can instantly eliminate them from your personal local browsing experience using uBlacklist.

What Problem Does uBlacklist Solve?

uBlacklist is a lightweight browser extension that filters specified sites from appearing in your search engine results.

Instead of landing on low-quality scraper sites or clickbait doorway networks:

Hides Spam Results: Blacklisted domains are automatically stripped or dimmed out directly in your search results.
Multi-Search Engine Support: Works seamlessly across Google, Bing, DuckDuckGo, Ecosia, Startpage, and more.
Community Rule Subscriptions: Instead of manually blocking sites one by one, you can subscribe to crowdsourced community blocklists that sync automatically in the background.

Step 1: How to Install uBlacklist on Chrome

Installing the extension takes less than a minute:

Open Google Chrome.
Visit the uBlacklist page on the Chrome Web Store.
Click Add to Chrome and confirm by clicking Add extension.
(Optional) Click the puzzle piece icon (Extensions) in your top-right toolbar and pin uBlacklist for easy access.

Step 2: Recommended Community Rule Sets

To get maximum protection without needing to build your own list from scratch, subscribe to curated community rule sets listed on the uBlacklist Rule Sets Directory.

I specifically recommend enabling these three subscriptions to target common web spam patterns:

Spam: Ad and tracking server domain list
- Purpose: Blocks domain networks known for aggressive popups, tracking traps, and ad-heavy doorway pages.
Spam: DGA & Random Subdomain Spam Blocklist
- Purpose: Targets programmatically generated, short-lived domains (e.g., random character strings like lwhiswfnuuiu.gay or ohgssoigkcik.icu) frequently used by automated web spam rings.
Copycat Sites: uBlock-Origin-dev-filter
- Purpose: Shields developers from scraper sites that duplicate StackOverflow, GitHub, and technical documentation with low-quality, ad-stuffed overlays.

Step 3: How to Add Recommended Rule Sets to Your Extension

Before subscribing directly from the community directory, enable subscriptions in your local extension settings:

Enable Subscriptions:
- Click the Extensions icon (puzzle piece) in your Chrome toolbar.
- Find uBlacklist, click the three dots ("..."), and select Options.
- Scroll down to the Subscriptions section and make sure the Enable subscriptions toggle is switched ON (Enabled).
Subscribe from the Community Directory:
- Visit the uBlacklist Rule Sets Directory.
- Locate the recommended rule sets:
  - Spam: Ad and tracking server domain list
  - Spam: DGA & Random Subdomain Spam Blocklist
  - Copycat Sites: uBlock-Origin-dev-filter
- Click the "+" (Subscribe) button next to each rule set.
Confirm:
- uBlacklist will prompt you to confirm—click Subscribe (or Add) to finish adding the list.

Step 4: How to Verify It’s Working

To ensure your rule sets are active and working as expected:

Check Subscription Status:
- Click the Extensions icon (puzzle piece) in your Chrome toolbar.
- Locate uBlacklist in the list, click the three dots ("...") next to it, and select Options.
- Scroll down to the Subscriptions section.
- You should see a status or timestamp showing "Updated [time]" alongside the total count of loaded rules/patterns for each subscribed list.
- You can click Update now to manually trigger an initial sync.
Test in Search Results:
- Perform a search query on Google or DuckDuckGo.
- If a domain matching your blocklists appears in the raw index, uBlacklist will intercept it.
- Look at the bottom or top of your search results page—you will see a subtle indicator stating: "uBlacklist has blocked N site(s)". Click it to view or temporarily unhide blocked results.

By pairing direct reporting with local client-side filtering like uBlacklist, you get an immediate, pristine search experience free from automated spam networks.

How to Build a Repeatable AI Search Visibility Benchmark

Tue, 08 Sep 2026 13:42:54 +0000

How to Build a Repeatable AI Search Visibility Benchmark

By Jarno Saarimies, Founder of AEOvara · Last updated: September 8, 2026 · 9 min read

Your buyers are already asking ChatGPT, Perplexity, and Google AI Overviews who to trust. Google says AI Overviews now reach 2 billion monthly users, and industry tracking puts them on roughly 48% of all queries — up 58% year over year (BrightEdge, February 2026). Meanwhile, Seer Interactive's longitudinal study found organic CTR on AI Overview queries dropped from 1.76% to 0.61% between June 2024 and September 2025.

Here's the problem: your analytics stack is blind to all of it. If ChatGPT recommends a competitor and never mentions you, nothing gets logged. No click, no impression, no bounce rate. The opportunity just evaporates.

A benchmark fixes this. Not a one-off audit you run once and forget — a repeatable measurement system that tells you, month after month, whether you're becoming more or less visible inside AI answers.

This guide shows you exactly how to build one, starting with a free spreadsheet method and scaling to automated tracking when the data demands it.

What "AI search visibility" actually means (define it before you measure it)

Before you measure anything, agree on what you're measuring. Five metrics carry the weight:

Mention rate — the percentage of tested prompts where your brand appears at all. This is your headline number.
Share of Voice (SoV) — your mentions as a percentage of all tracked-brand mentions in the same answers. Peec AI's own documentation shows why this matters: a brand mentioned 4 times against a competitor's 12 mentions has high visibility but only 25% SoV. Visibility is presence. SoV is competitive prominence.
Citation rate — whether the answer actually links to you, not just names you. Track this separately: on ChatGPT most mentions carry no link, while Perplexity links almost everything.
Position and prominence — where you land in the answer. AI engines tend to treat the first-named brand as the default recommendation. Buried in the last paragraph is not the same as leading the list.
Sentiment and accuracy — not just whether you appear, but how you're described. A confident but outdated mention (wrong pricing, wrong positioning) can cost you the deal.

Step 1: Build a frozen prompt corpus

Pick 10–15 prompts that real buyers would actually ask. Not keywords — questions. Organize them into five categories:

Prompt type	What it reveals	Template
Category / discovery	Whether you're in the consideration set at all	"What are the best [category] for [use case]?"
Comparison	Positioning against a named rival	"How does [your brand] compare to [competitor]?"
Persona / use-case	Whether you're recommended in your buyer's context	"What should a [role] look for when choosing a [category]?"
Problem / jobs-to-be-done	Top-of-funnel presence before your category is named	"How do I [problem]?"
Brand-direct / reputation	Accuracy and sentiment	"What is [your brand]?" / "Is [your brand] any good?"

This list stays mostly frozen. This matters more than most people realize. If you change the prompts every month, you're not tracking visibility — you're running random tests. You can add prompts once a quarter, but the core set must stay stable or your trend data is meaningless.

Step 2: Choose your engines and cadence

Measure each platform separately, because a strong ChatGPT score tells you nothing about Perplexity or AI Overviews. At minimum: ChatGPT, Perplexity, Gemini, and Google Search with AI Overviews.

On cadence: weekly tracking is the standard for most teams (LLM Pulse and similar platforms default to this). Daily tracking makes sense for product launches or volatile categories — finance shows the highest citation volatility of any vertical. Monthly is the absolute floor for a manual process.

Run each prompt 3–5 times per engine. Generative answers are probabilistic — the same question names different brands on different runs. One query is an anecdote, not a measurement. Research on robust benchmarks suggests 500–2,000 prompts per week for statistically stable values (SUMAX Research, 2026); for a manual baseline, 10–15 prompts × multiple runs × 3–4 engines is a realistic starting point.

Step 3: Run the manual baseline (this afternoon, free)

You don't need a platform yet. You need a spreadsheet and about an hour.

Run your prompt set through each engine — same wording, same account settings where possible. For each response, log:

Query text
Platform and date
Mentioned? (Yes/No)
Position (1st named, 2nd, 3rd…)
Cited with a link? (Yes/No)
Competitors named
Sources cited in the answer
Accuracy/sentiment notes
Screenshot

Screenshot everything. AI results move around. Something that shows up this month may disappear next month. If you want to understand trends, you need receipts.

The last column is usually the most useful. If Perplexity or another engine shows sources, those sources tell you where the answer gets its confidence. Sometimes the fix isn't "write another blog post" — it's getting your company described better on the third-party pages the AI is already reading.

Step 4: Calculate your benchmark numbers

Four formulas turn raw logs into a benchmark:

Mention rate = prompts where you appear ÷ total prompts run × 100
Citation rate = prompts with a link to you ÷ total prompts × 100
Share of Voice = your mentions ÷ all tracked-brand mentions × 100
Position score = average position when mentioned (lower is better)

Example: mentioned in 5 of 12 prompts on Perplexity, with 4 of those including a link. Mention rate: 42%. Citation rate: 33%. Your competitor mentioned in 9 of 12 with 7 links. SoV: 5 ÷ 14 = 36%.

That gives you a real baseline — and usually a few uncomfortable surprises. One thing to internalize early: position in a single answer is mostly noise; frequency across many runs is the signal (as SparkToro's analysis showed). Track how often you appear, not where you landed once.

Step 5: Add context metrics the spreadsheet can't see

A prompt log tells you that you're visible or invisible. Three additional signals tell you whether it matters:

AI referral traffic. Expect attribution to undercount. Add a "How did you hear about us?" option with an AI assistant choice to your intake forms, because most AI mentions never produce a trackable click.

Branded search lift. Growth in people searching your company name often correlates with AI exposure, per Percepture's GEO research.

Conversion quality. The traffic you do get is disproportionately valuable: Ahrefs' own data showed AI-referred visitors converting at 23x the rate of traditional organic visitors (0.5% of traffic generated 12.1% of signups, June 2025). Semrush found a similar 4.4x pattern. Small numbers, high intent.

Step 6: Automate when manual stops scaling

Twenty-five prompts run five times each is 125 data points — enough for a baseline, not enough for stable rates. That's when a platform earns its money. Roughly priced landscape as of mid-2026:

Otterly AI — from $29/month (15 prompts). The low-cost serious entry point; weekly refresh.
Peec AI — from $95/month (50 prompts). Clean visibility, position, sentiment, and SoV dashboards; strong for structured reporting.
Semrush AI Visibility Toolkit — from $99/month standalone. Low friction if you're already on Semrush.
Ahrefs Brand Radar — from $199/month per AI-platform index (requires an Ahrefs base plan from $129/month). Prompt methodology anchored to search data.
Profound — enterprise positioning, deep analytics, most advanced features behind custom pricing.

Whichever tool you pick, demand these capabilities: multi-platform tracking (ChatGPT, Perplexity, Gemini, AI Overviews, Claude), citation/source URL analysis — not just mention counts — competitor benchmarking, historical trends, and transparency about prompt volume methodology.

The seven mistakes that break benchmarks

Chasing rank position instead of visibility rate. Position swings wildly run to run; frequency is the stable metric.
Measuring only one engine. You're optimizing blind.
Counting mentions but ignoring citations (or vice versa). They're different metrics doing different jobs.
Ignoring sentiment and accuracy. Being named with wrong pricing isn't a win.
Running each prompt once. Anecdote, not measurement.
Expecting clean click attribution. Analytics will always undercount AI's influence.
Testing once and moving on. Visibility drifts as models update and content ages. Only a fixed cadence reveals what's actually moving.

The bottom line

You can't manage what you can't see, and traditional analytics can't see AI answers. The process is unglamorous: freeze a prompt set, run it repeatedly across engines, log mentions and citations separately, review monthly, automate when the volume outgrows your spreadsheet. Do that consistently and "how visible are we in AI?" stops being a worry and becomes a number you can move.

FAQ

What is an AI search visibility benchmark?
An AI search visibility benchmark is a fixed set of buyer-relevant prompts that you run repeatedly across AI search engines (ChatGPT, Perplexity, Gemini, Google AI Overviews) to measure how often and how accurately your brand appears in AI-generated answers over time. It turns "are we visible in AI?" from a vague worry into a trackable number with a trend line.

How many prompts do I need for a reliable benchmark?
For a manual baseline, 10–15 prompts across five categories (category, comparison, persona, problem, brand-direct), run 3–5 times per engine, gives directional data. For statistically stable visibility rates, research suggests 500–2,000 prompts per week — which is where automated platforms become necessary.

What's the difference between mentions and citations?
A mention means the AI names your brand in the answer text. A citation means it links to your website as a source. Mentions build brand association; citations drive trackable traffic. On ChatGPT most mentions carry no link, while Perplexity links almost everything, so the two metrics must be tracked separately.

How often should I re-run my benchmark?
Weekly is the standard cadence for automated tracking; monthly is the minimum for manual runs. Re-run the exact same prompt set each time — changing prompts between runs destroys comparability. Also re-baseline after major model updates, since visibility can shift overnight when models change.

Can I build an AI visibility benchmark for free?
Yes. A spreadsheet, 10–15 frozen prompts, and one hour a month covers the fundamentals — mention rate, citation rate, position, competitors, and cited sources. Paid tools earn their cost when you need scale, daily cadence, sentiment scoring, and source-level analysis across hundreds of prompts.

Do I still need traditional SEO tracking?
Yes. Ranking on page one remains correlated with AI citations — Ahrefs found 38% of AI Overview-cited pages also rank in the top 10 — but the relationship is weakening fast (down from ~76% in mid-2024). The two systems measure different things: blue-link position versus presence inside the answer. You need both.

Jarno Saarimies is the founder of AEOvara, a Finnish consultancy specializing in AI search optimization (AEO/GEO/LLMO) — helping brands get found, cited, and recommended inside AI-generated answers. A professional photographer since 2013, Jarno runs Kuvaajankulma, a photography studio in Lappeenranta, Finland, and applies the same evidence-first mindset to search: no hype, no guesswork — measure what actually matters. AEOvara Jarno S
AEOvara BLOG

Google Search Console Setup for Next.js (2026 Beginner's Guide)

Sun, 16 Aug 2026 05:54:15 +0000

You just shipped your Next.js site. Metadata's wired up, Open Graph tags look great, JSON-LD is in place — and yet Google has no idea your site exists. Weeks go by, you check your analytics, and organic traffic is basically zero.

Here's the thing nobody tells beginners: writing good SEO code isn't the same as telling Google to actually crawl it. That connection happens through one tool — Google Search Console (GSC) — and it's one of the most important tools new Next.js developers should set up after launch.

By the end of this guide, you'll have Google Search Console fully connected to your Next.js 16 App Router site, a submitted sitemap, a correctly configured robots.txt, and a repeatable process for catching indexing problems before they cost you traffic.

Why This Is Tricky

Google Search Console isn't part of your codebase, so it's easy to treat it as an afterthought — something you'll "get to later." But GSC is the only place that tells you:

Whether Google can actually crawl your pages
Which pages are indexed (showing up in search) versus silently ignored
Real search queries people used to land on your site
Core Web Vitals issues that can contribute to a poor page experience and potentially affect search performance

Without it, you're optimizing blind. You could have a metadata bug shipping noindex to your entire site for a week and never know until traffic craters.

Prerequisites

This guide assumes:

Next.js 16 with the App Router
TypeScript in strict mode
A site already deployed to a live domain (Vercel, Netlify, or self-hosted — the Next.js side works the same everywhere)
Access to your domain's DNS settings (through your registrar or host)

If your project doesn't have app/sitemap.ts or app/robots.ts yet, don't worry — we'll build both from scratch.

Step 1: Create the Google Search Console Property

Head to search.google.com/search-console and sign in with the Google account you want to permanently own this property. If you're doing this for a client, use an account you control and add them as a verified owner afterward — don't set the property up under their personal Gmail unless you're handing off the project entirely.

You'll be asked to choose a property type:

Property type	Covers	Verification
Domain property	All subdomains and both `http`/`https` versions under one property	DNS only
URL-prefix property	Just the exact URL pattern you enter	DNS, HTML file, meta tag, GA, or GTM

💡 Tip: For a new site, go with a Domain property. It's the most complete option — you get www and non-www, http and https, and every subdomain in a single dashboard instead of juggling separate properties for each variant.

The tradeoff is that Domain properties only support DNS verification, which means you need access to your domain's DNS zone (through your registrar, or through Vercel's domain settings if Vercel manages your DNS).

Step 2: Verify Ownership

Option A — DNS TXT Record (Recommended for Domain Properties)

This is the most reliable method because it's tied to the domain itself, not a specific file or page — it survives redeploys, framework migrations, and hosting changes.

GSC gives you a TXT record value that looks like google-site-verification=abc123...
Log into your DNS provider (Cloudflare, Namecheap, GoDaddy, or your registrar)
Add a new TXT record:
- Host/Name: @ (represents your root domain)
- Value: the code GSC gave you
- TTL: default (usually 3600)
Save, go back to GSC, and click Verify

⚠️ Common Mistake: DNS changes aren't instant. Some providers (Cloudflare especially) propagate in 2–3 minutes, but others can take up to an hour. If verification fails immediately, wait and retry rather than assuming you did something wrong.

Option B — HTML Meta Tag (URL-Prefix Properties)

If you went with a URL-prefix property instead, GSC gives you a meta tag like:

<meta name="google-site-verification" content="your-code-here" />

In Next.js's App Router, don't paste this directly into a <head> tag — the Metadata API has a dedicated field for it:

// app/layout.tsx
import type { Metadata } from "next";

export const metadata: Metadata = {
  // ...your existing metadata
  verification: {
    google: "your-code-here",
  },
};

What's happening here: Next.js's Metadata API generates the correct <meta name="google-site-verification"> tag for you, merged in with the rest of your root layout's metadata. This is safer than hand-editing HTML because it can't accidentally get wiped out by a template change.

⚠️ Common Mistake: Only setting the verification field in a nested page's metadata instead of the root app/layout.tsx. Google checks your homepage for this tag, so it has to live in metadata that actually applies to /.

Redeploy, then click Verify in GSC.

Step 3: Generate a Sitemap with `sitemap.ts`

A sitemap is a map of every URL you want indexed. It doesn't force Google to index anything, but it can make URL discovery more efficient — especially for new sites, large sites, or pages with few internal links.

Next.js has a built-in file convention for this — no packages needed:

// app/sitemap.ts
import { MetadataRoute } from "next";
import { getAllPosts } from "@/lib/posts";
import { appURL } from "@/lib/utils";

export default function sitemap(): MetadataRoute.Sitemap {
  const posts = getAllPosts().map((post) => ({
    url: `${appURL}/blog/${post.slug}`,
    lastModified: new Date(post.updatedAt || post.publishedAt),
  }));

  const staticRoutes = ["", "/about", "/blog", "/contact"].map((route) => ({
    url: `${appURL}${route}`,
  }));

  return [...staticRoutes, ...posts];
}

What's happening here: dropping a file named sitemap.ts inside app/ tells Next.js to automatically generate a working route at /sitemap.xml — you never manually build or serve XML. lastModified tells search engines when a page was last meaningfully updated. Google can use an accurate lastmod value when deciding when to recrawl a URL. Although Next.js supports changeFrequency and priority, Google says it ignores those two sitemap values, so don't treat them as ranking or crawling controls.

💡 Tip: Only include canonical URLs that you actually want search engines to discover and potentially index. Don't add /admin, /api/*, private pages, draft URLs, or other non-indexable routes to your sitemap.

Step 4: Configure `robots.ts` Correctly

robots.txt tells crawlers which URLs they can or can't request. It controls crawling — not whether a page can appear in search results.

// app/robots.ts
import { MetadataRoute } from "next";
import { appURL } from "@/lib/utils";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: "*",
      allow: "/",
      disallow: ["/admin", "/api"],
    },
    sitemap: `${appURL}/sitemap.xml`,
  };
}

What's happening here: same pattern as the sitemap — Next.js auto-generates a working /robots.txt from this file. allow: "/" lets crawlers visit everything by default, disallow blocks specific internal paths, and pointing sitemap at your sitemap URL here means crawlers can discover it even before you manually submit it in GSC.

⚠️ Common Mistake: Leaving disallow: "/" in robots.ts after moving from a staging environment to production. This single line blocks your entire site from every search engine. Always double-check this file immediately after your first production deploy.

Tip If you need to prevent a page from being indexed, use noindex rather than relying on robots.txt.

Step 5: Submit Your Sitemap in GSC

Once you're verified:

In the GSC sidebar, click Sitemaps
Enter sitemap.xml (GSC prepends your domain automatically)
Click Submit

You should see a "Success" status within a few minutes, though it can take a day or two before "Discovered URLs" starts populating.

💡 Tip: If your sitemap shows "Couldn't fetch," visit yourdomain.com/sitemap.xml directly in your browser first. If it 404s, the issue is your deployment, not GSC — check that app/sitemap.ts actually made it into your production build.

Step 6: Verify Robots Meta Tags Aren't Blocking Individual Pages

robots.txt controls crawling at the site level, but an individual page can still say "don't index me" through page-level metadata — easy to forget on a page you copy-pasted from a draft.

// app/some-page/page.tsx
import type { Metadata } from "next";

export const metadata: Metadata = {
  robots: {
    index: false,
    follow: false,
  },
};

⚠️ Common Mistake: Setting index: false on the root layout "just to be safe" during development, then forgetting to flip it before launch. This is one of the most common reasons a brand-new Next.js site gets zero organic traffic for weeks — always check your homepage's rendered <meta name="robots"> tag in production before assuming GSC is the problem.

You normally don't need to explicitly set index: true and follow: true. The important thing is making sure you haven't accidentally configured these false on pages you want in search.

You can check this instantly using GSC's URL Inspection Tool (covered next) — it tells you directly whether Google sees a page as indexable.

Step 7: Use URL Inspection to Force Indexing

The URL Inspection Tool is the single most useful feature in GSC day-to-day. Paste any URL from your site into the search bar at the top of GSC, and it tells you:

Whether the page is currently indexed
When it was last crawled
Whether anything (robots meta, canonical mismatch, noindex) is blocking it

After publishing a new page or fixing a metadata bug, you can use Request Indexing to ask Google to recrawl the URL. It doesn't guarantee immediate crawling or indexing, so use your sitemap for ongoing URL discovery rather than repeatedly requesting individual URLs.

💡 Tip: Request Indexing has a daily quota per property. Don't use it for bulk resubmission of your whole site — that's what your sitemap is for. Reserve it for genuinely new or recently fixed pages.

Step 8: Check Core Web Vitals

GSC's Core Web Vitals report shows real-user field data (from actual Chrome visitors), not just a local Lighthouse score. This matters because Core Web Vitals are used by Google's ranking systems, and lab data can look great locally while real users on slower connections or devices have a worse experience.

Watch for pages slipping from "Good" into "Needs improvement" or "Poor" — this is usually caused by:

Unoptimized images (missing next/image, or missing width/height)
Large third-party scripts loaded without next/script strategy
Web fonts causing layout shift instead of using next/font

The report uses real-user data rather than a synthetic Lighthouse test, so your local performance score and your users' actual experience can tell different stories.

Related: Image Optimization in Next.js 16 covers the next/image configuration options that most directly affect this report.

Step 9: Check Your Canonical URLs

Google doesn't only decide whether a page can be indexed — it also chooses which URL represents duplicate or very similar versions of the same content.

In Next.js, make sure important pages have the correct canonical URL, especially if your site supports multiple URL variations.

import type { Metadata } from "next";

export const metadata: Metadata = {
  alternates: {
    canonical: "https://example.com/blog/my-post",
  },
};

⚠️ A canonical URL is a hint, not a guarantee. Google can choose a different canonical if it determines another URL is a better representative of the content.

Step 10: Understanding "Discovered — Currently Not Indexed"

Sooner or later you'll open the Page indexing report and see a bucket of URLs labeled Discovered — currently not indexed. This is one of the most common statuses beginners panic over, so it's worth understanding what it actually means.

It means Google found the URL — usually through your sitemap or an internal link — but hasn't crawled and indexed it yet. This is different from Crawled — currently not indexed, where Google did visit the page but chose not to index it. Both statuses are common and don't always signal a problem; a brand-new site with dozens of freshly submitted URLs will often show a pile of "Discovered" pages that clear up on their own within a couple of weeks.

That said, if a URL sits in this state for more than two to three weeks, it's worth investigating:

Thin or low-value content — tag pages, barely-filled pages, or near-duplicates may provide little unique value. Make sure important URLs contain useful, distinct content.
Crawl budget — very large sites, or sites publishing a lot of near-identical pages quickly, can outpace how much Google is willing to crawl in a given period.
Weak internal linking — a page with no other pages linking to it signals lower importance. Link to new posts from your homepage, a category page, or related content shortly after publishing.
Server response issues — if Googlebot hit a slow response or a temporary error when it tried to crawl, it may deprioritize a retry rather than fail loudly.

💡 Tip: Don't spam Request Indexing on the same "Discovered" URL every day — it doesn't speed anything up and doesn't change why Google deprioritized the crawl in the first place. Fix the underlying content or linking issue, then request indexing once.

Step 11: Don't Forget Bing Webmaster Tools

Google isn't the only crawler worth your time. Bing powers Microsoft's search ecosystem and is also an important source of web search visibility beyond Google — real, easy traffic most developers skip entirely.

Sign up at Bing Webmaster Tools
Submit the exact same sitemap.xml you already built — no separate file needed
Bing lets you import your verified site directly from Google Search Console in a couple of clicks, skipping manual re-verification entirely

A Simple Weekly Routine

Set aside 10–15 minutes a week and check:

Performance report — any unusual drops or spikes in clicks/impressions?
Page indexing report — any pages stuck in "Discovered — currently not indexed" for more than two weeks?
Core Web Vitals — any pages that moved from Good to Needs Improvement?

Catching a regression here in week one is a five-minute fix. Catching it three months later, after it's quietly tanked your rankings, is a much longer recovery.

Troubleshooting Checklist

If something isn't working, work through these in order before assuming it's a GSC bug:

[ ] Verification failing? Wait at least 30–60 minutes after adding a DNS TXT record before retrying — propagation isn't instant.
[ ] Verification meta tag "not found"? View your production page source (not localhost) and confirm the google-site-verification tag is actually in the rendered <head> — check it's set in app/layout.tsx, not a nested page.
[ ] Sitemap "Couldn't fetch"? Visit yourdomain.com/sitemap.xml directly in a browser first. A 404 means a deployment problem, not a GSC problem.
[ ] Sitemap fetches but pages aren't indexed? Check robots.ts isn't disallowing the path, and check the page's own metadata isn't setting robots: { index: false }.
[ ] Whole site missing from search? Check app/robots.ts for an accidental disallow: "/" — this is the single most common self-inflicted outage.
[ ] Pages stuck on "Discovered — currently not indexed"? Give it 2–3 weeks first. If it persists, check for thin content and add internal links pointing to the page.
[ ] Core Web Vitals showing "Poor"? Confirm images use next/image, fonts use next/font, and large third-party scripts use next/script with an appropriate loading strategy.
[ ] Data missing entirely after adding the property? This can be normal for a new property. Google says data collection begins when the property is added, even before verification, but it can take up to a week for data to appear.

Frequently Asked Questions

Do I need a Domain property or URL-prefix property for Next.js?

Domain property, in almost every case — it covers www, non-www, and both protocols under one dashboard. Only use URL-prefix if you specifically need isolated data for a subdomain or can't get DNS access.

How long until I see data in Google Search Console?

Search Console doesn't give you historical performance data from before Google started collecting data for the property. Google says data collection can begin when the property is added, even before ownership verification is completed. In practice, expect the reports to take some time to populate, and don't panic if impressions stay low for the first couple of weeks on a brand-new site.

My sitemap shows "Couldn't fetch." What's wrong?

Almost always a deployment issue, not a GSC issue. Visit yourdomain.com/sitemap.xml directly — if it 404s or shows an error, check that app/sitemap.ts is actually included in your production build and isn't being blocked by robots.ts.

Can I use both a DNS TXT record and a meta tag verification?

Yes. You can maintain multiple verification methods. This can be useful as a backup, although you don't need multiple methods for normal operation. Domain properties use DNS verification, while URL-prefix properties support additional methods such as HTML meta tags.

Does adding a sitemap guarantee my pages get indexed?

No. A sitemap only helps Google discover URLs faster — it doesn't override noindex tags, thin-content penalties, or crawl budget limits. Pages still need to pass Google's own quality bar to actually rank.

Wrapping Up

Google Search Console setup for Next.js comes down to five things that take about 20 minutes total: verify ownership, ship app/sitemap.ts, ship a correctly scoped app/robots.ts, submit the sitemap, and build a weekly habit of checking the Performance and Page Indexing reports. None of it is complicated — it's just easy to skip, and skipping it is exactly why so many well-built Next.js sites sit invisible in search for months after launch.

From here, a natural next step is pairing this with a proper JSON-LD structured data setup so your indexed pages are eligible for rich results, not just a plain blue link.

Continue Learning

Free SEO Tools

JSON-LD Generator for Next.js & HTML - generate Article, FAQ, Product, LocalBusiness, SoftwareApplication, Breadcrumb, Person, Organization, and Website JSON-LD — checked live against Google's actual Rich Results requirements.
Meta Tag Generator for Next.js & HTML — generate SEO meta tags, Open Graph tags, Twitter Cards, and canonical URLs instantly. Handy for wiring up per-post metadata like the generateMetadata() function above.
Supabase RLS Policy Generator — if your next project pairs this blog setup with a Supabase backend, generate secure, performance-optimized Row Level Security policies in seconds.
Zod Schema Generator — generate Zod schemas instantly from JSON or TypeScript. Includes smart inference, live validation, React Hook Form, Next.js API Routes, and Server Actions.

I built an independent search engine—and the results were interesting

Sat, 06 Jun 2026 13:15:32 +0000

A few months ago, I came across FrogFind, a retro search engine, and I decided that I wanted to try my hand at making one, too.

I first started with the crawler, carefully planning out a simple single threaded Python requests bot using SQLite3 that accommodated robots.txt. This was particularly difficult—I had barely any experience in the field of search engine creation, and its subcategories.

After the crawler was the webapp, which was relatively easy, because of my experience with Flask. I used Redis for rate limit storage, and various algorithms to do the searching for me.

I shoved my crawler and webapp into a pip package, quokko, with a CLI to control it. I wanted my project to be accessible, and for people to create their own search engines without trouble.

Hosting it, as of Jun 6, 2026, there are around 13 million entries, which is very little for a search engine. The engine's quality may not be ideal, but it seems to provide me all these curious corners of the Internet, and also seems to be a tad more unfavourable to companies in the Big Tech category.

PROJECT HAS BEEN CLOSED DUE THE THE HASSLE OF HOSTING IT. . . SORRY!

It's now hosted here: quokko.splot.dev
And you'll find the code and documentation here: https://codeberg.org/splot-dev/quokko

This is a hobby project; errors are inevitable.

Its simple customizability allows for people to make their own micro search engines, and the goal is for these micro search engines to form a net of niche, curious alternative human knowledge collections.

The goal of Quokko is not to become a competitor search engine, but for it to be the open door for more other search engines.

Why Updating Data in Postgres Is More Expensive Than You Think

Tue, 20 Jan 2026 03:30:41 +0000

Context

I was recently tasked with optimizing stockinsights.ai’s full text search system for one of our core features: keyword search over filings.

Previously, this worked by storing entire documents as raw strings in MongoDB and scanning through them at query time to return surrounding snippets. As you can imagine, this approach became increasingly slow and inefficient as the number of filings kept growing.

This clearly needed to be fixed.

While looking at the existing system, I noticed that we already stored chunked text snippets in a separate Postgres instance as part of our embeddings tables for semantic search. Each chunk already existed as a row, paired with an embedding.

It seemed obvious that we should reuse this same chunked data for keyword search as well.

At a high level, the idea looked something like this:

Keyword search across snippets
→ return matching text chunks
→ rank documents higher if more chunks match
→ return one representative snippet per document

All I needed to do, or so I thought, was add another column to store tsvectors (Postgres’s internal data structure for full text search) and create a GIN index on top of it.

How hard could it be?

The obvious solution

Postgres has built-in full text search using tsvector and tsquery. The plan was straightforward:

Add a tsvector column to the embeddings table
Populate it from the existing text column
Create a GIN index
Query snippets directly instead of scanning raw documents

Note: We could have normalized this into a separate table, but at a startup, we often accept some denormalization in favor of simpler systems.

From an application point of view, this was almost trivial. No new data model, no new pipeline, just reuse what already existed.

The mistake was assuming that a small logical change would translate into a small physical change inside the database.

When the costs showed up

**
Initially, nothing failed. The migration just took much longer than expected.

Each document type table had roughly 2 to 5 million rows. Updating even 10,000 rows was taking hours, and each iteration made the next one slower. What started as a small backfill quickly turned into something that felt unbounded in cost.

Populating the new column dragged on, disk usage grew far beyond what the additional data alone could justify, and creating the GIN index noticeably impacted write throughput while it was running.

More concerning was that queries themselves began to slow down. Reads that were previously fast started degrading as the table and indexes bloated, even though the query plans hadn’t meaningfully changed.

At that point, it was clear that something deeper was going on.

UPDATE in Postgres is not mutation

Postgres stores tables as heap files. A heap file is an unordered collection of tuples, and tuples are immutable.

When you run an UPDATE, Postgres does not modify a row in place. Instead, it:

Inserts a new version of the row
Marks the old version as no longer visible
Leaves cleanup for later
This design exists to support MVCC (multi-version concurrency control), which allows readers and writers to operate concurrently by keeping multiple versions of rows and resolving visibility at read time.

In practice, this means UPDATE is not a mutation of an existing row, but the creation of a new version of that row.

Why adding a column rewrites every row

A Postgres row is stored as a single physical tuple containing all columns. What I learned here is that there is no column-level storage, and therefore no partial row update.

When you populate a new column on an existing table, Postgres must:

Copy every existing column value
Append the new column value
Write an entirely new tuple
Even if every other column remains unchanged, the entire row is rewritten. On a large table, this effectively means rewriting the entire dataset, not just adding one column.

Why text, tsvector, and embeddings make this worse
Our table already stored text snippets and embeddings, and we were about to add tsvectors on top of that.

A few things compound the cost here:

Text columns are variable length. Large values often exceed the inline storage limit. When that happens, Postgres uses TOAST.
TOAST stores oversized column values out of line and keeps pointers to them in the main tuple.

Even when data is TOASTed, updating the row still requires rewriting the main tuple and, in some cases, the TOAST entries themselves.

tsvector expands data. It is not just the original text. It stores lexemes (similar to tokens) along with positional metadata, and it is also typically stored via TOAST.

Embeddings make rows wide. Embeddings are large binary float arrays stored alongside text and indexed separately. Combined with text and tsvector, each row becomes wide, and wide rows are expensive to rewrite.

This combination pushes right up against the limits of what an immutable row store is comfortable handling.

Indexes amplify the cost

On top of this, the table already had indexes.

Every new tuple version requires index maintenance. For each index, Postgres must:

Insert new index entries
Maintain index structure
Write corresponding WAL records
GIN indexes are especially heavy. A single row can generate many index entries, often one per token in the tsvector.

While most indexes in Postgres are B-tree based, GIN indexes are inverted indexes.

WAL (write-ahead log) is the sequential log used for crash recovery and replication. More index work means more WAL traffic, which directly affects disk usage, replication lag, and write latency.

Indexes make reads fast, but they multiply write cost. That tradeoff is unavoidable.

An alternate approach: COPY, swap, drop

While digging through Postgres blogs and discussions, I came across an alternative approach for modifying very large tables.

Instead of updating the table in place, you:

Create a new table with the desired schema
Copy data into it
Recreate indexes
Swap it with the old table
Drop the old table
The COPY-swap-drop approach.

This has a few important properties:

Each row is written exactly once
No dead tuples are created
Indexes are built cleanly
There is no incremental index maintenance during updates
By contrast, UPDATE creates deferred work. Dead tuples accumulate and must be cleaned up later, and indexes are updated repeatedly for each new row version.

In an immutable storage model, rewriting data cleanly can often be cheaper than mutating it incrementally.

Final design

One additional factor that helped was simply scaling the RDS instance. More memory meant more rows could be processed efficiently during the rebuild, which significantly reduced runtime.

The final workflow looked like this:

Upgrade the instance
Use COPY-swap-drop to rebuild the table
Recreate indexes
This approach ended up being far faster and more predictable than trying to UPDATE rows in place.

SEO for Vibe Coders: Key Strategies to Get Your AI-Crafted Tools Discovered

Tue, 12 Aug 2025 23:03:06 +0000

“Vibe coding” is revolutionizing software development. Instead of manually writing code, developers now use natural language prompts to guide AI tools—like GitHub Copilot, Cursor, or Bolt—toward functional code generation. While it accelerates deployment, it increases the challenge of making code or tools discoverable via search.

Search Engine Optimisation for Vibe Coders

This guide shows how vibe coders can utilise SEO techniques to ensure their AI-generated tools, scripts, or portfolios surface in search—even as traditional search evolves toward AI-managed answers.

What Is Vibe Coding & Why SEO Needs to Adapt?

Coined by Andrej Karpathy in 2025, vibe coding emphasizes using AI to generate code from conversational prompts. It accelerates development but also risks creating obscure or undocumented tools that traditional search engines struggle to index effectively.

Meanwhile, the search landscape is shifting toward AI Answer Engines—platforms like ChatGPT, DeepSeek, Google’s AI Overview, and Bing AI that deliver answers directly without requiring a click. Visibility now demands not just ranking, but being cited within AI responses.

Also Read, OpenAI’s GPT-5—now live—unifies advanced reasoning with multimodal capabilities and massive context

SEO Strategies Vibe Coders Should Implement

Example SEO-Christian Workflow

Project Page: Create a page with structured URL like /vibe-coder-todo-app and title “Vibe-Coded To-Do App with AI Guidance.”
Documentation: Add a blog-style walkthrough titled “How I Built My To-Do App Using Vibe Coding”—explain prompts, iterations, debugging tips.
FAQ Section: Insert Q&A such as “What is vibe coding?” or “Why choose AI-over-manual coding?” and include FAQ schema.
Schema Markup: Use SoftwareSourceCode and FAQPage in JSON-LD to help AI systems recognize your content.
Performance Boost: Optimize images, scripts, and rethink mobile behavior—boost Core Web Vitals.
Track Visibility: Monitor Google Analytics and look for AI-driven features like featured snippets or answer panels.

Why This Mindset Is Important?

Generative SEO is the future: Being discoverable isn’t just about keywords—it’s about being cited by AI systems.
Vibe Coding Democratizes Development: But without SEO, your contributions may remain invisible. The right optimization brings visibility.
Sustain Relevance: As AI reduces the need for clicks, preparing your content for voice search, snippet, and AI inclusion maximizes reach.

Also Read, GEO and AEO redefine SEO for 2025 by structuring content to be cited—rather than just ranked—by AI!

Concluding Thoughts

Vibe coders lead the next wave of AI-powered engineering—but keeping your projects discoverable requires a new SEO approach. You should focus on documentation, structured data, conversational-focused content, and performance.

Ready to audit your vibe-code portfolio or set up schema for AI visibility? Drop me a line—I’d love to help you get found by AI agents and humans alike.

DevLog 20250523: Sitemap and `robots.txt`

Wed, 18 Jun 2025 10:19:32 +0000

Search engine optimization (SEO) is not just about keywords and HTML metadata! Though those are the most basic things one can do (and can easily improve site visibility), there are other tricks that go a bit deeper—more technical than what ordinary readers see.

I found that webmasters can go to Google Search Console and Bing Webmaster Tools to add sitemaps. This continues our previous discussion on Search Engine Architecture.

Sitemap

A sitemap is simply a file (or sometimes a web page) that tells search engines about the pages on our site.

Better discovery: Search engines won’t have to “guess” at pages.
Faster indexing: New or updated content gets found more quickly when we update the sitemap.
Structured hints: Metadata in an XML sitemap gives crawlers extra clues about how often and how important different pages are.

There are two main flavors:

XML Sitemap (for search engines)
- It’s an XML-formatted file (usually named sitemap.xml) that lives at the website’s root.
- Inside, it lists all of the website’s important URLs, plus optional metadata like:
  - <lastmod> (when the page was last changed)
  - <changefreq> (how often it tends to be updated)
  - <priority> (a hint about which pages we consider most important)
- By submitting this file to Google Search Console or Bing Webmaster Tools, we help crawlers discover and index our pages more efficiently—especially useful if we have a very large site, pages that aren’t well linked internally, or lots of media content.
```
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/</loc>
    <lastmod>2025-05-20</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://www.example.com/blog/post-1</loc>
    <lastmod>2025-05-18</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
  
</urlset>
```
HTML Sitemap (for people)
- It’s just a regular web page on the site that lists links to all pages in a human-readable format.
- It’s primarily a usability feature—helping visitors (and indirectly search engines) navigate large or complex sites.

robots.txt

The robots.txt file is another “cheat-sheet” to put at the very root of the website (e.g. https://www.example.com/robots.txt) to tell well-behaved web crawlers which parts of the site they’re welcome to explore—and which parts we’d rather keep off-limits.

Privacy & security: Keep staging directories, admin panels, or confidential files out of search results.
Crawl-budget control: On large sites, we can steer crawlers away from low-value pages (like faceted filters), so they focus on important content.
Performance: Reduce server load by preventing bots from hammering resource-heavy sections.

Some notes:

Location matters
- Must live at https://domain.com/robots.txt (exactly).
- Crawlers automatically look here first before they begin crawling website pages.

Basic syntax

It’s plain text, with directives grouped by User-agent (the crawler’s name).
Common directives:
- Disallow: — path (or file) we don’t want crawled
- Allow: — exception to a Disallow: (supported by Google, Bing, etc.)
- Sitemap: — URL of the XML sitemap

# Block all crawlers from /private/
User-agent: *
Disallow: /private/

# Allow Googlebot to see /private/public-info.html
User-agent: Googlebot
Allow: /private/public-info.html

# Let everyone know where the sitemap lives
Sitemap: https://www.example.com/sitemap.xml

User-agents
- * is the wildcard: applies to every crawler.
- We can target specific bots (e.g., User-agent: Googlebot, User-agent: Bingbot) if we need different rules.
Disallow vs. Allow
- Disallow: / — don’t crawl anything on the site.
- Disallow: (empty) — allow everything.
- Allow: /path/to/page.html — lets a crawler index a page that would otherwise be blocked by a broader Disallow:.

A few best practices

robots.txt is publicly visible—don’t use it to hide truly sensitive info (use authentication!).

Test in Google Search Console’s “robots.txt Tester” or Bing Webmaster Tools.
Combine with sitemaps: Always include a Sitemap: line so crawlers can discover all valid URLs easily.

Here is an example from Google: https://www.google.com/robots.txt

DevLog 20250520: Search Engine Architecture

Wed, 21 May 2025 01:14:38 +0000

Overview

For an internal database, we needed to implement some sort of text-based search capabilities to help users locate documents. One goal was to be lightweight, hassle-free, easy to set up, expand, and maintain—and ideally, to set it up once and forget about it. It's okay if it costs a bit of computing power, but agility, quick iterations, usability, and robustness are of top concern.

Summary

It's surprising how much we can do without a server. Most serverless or static webpage hosts are happy to serve anything from a few MB to a few GB for free, likely even distributed across the globe. For any particular individual, if the search experience is truly customized to their needs, then not much resource is needed to deliver a satisfying experience.

Let's illustrate this with an example. Suppose that, for any webpage, the information we care about includes the following:

Title
Abstract
URL
Keywords (weighted)
Tags
Last Modification Time
Webpage Language

If we assume the abstract is at most 200 words and the rest adds up to 50 bytes, then each website we index takes about 250 bytes to store. 1KB fits four entries, 1MB fits 4096 entries, and 3MB fits about 12K entries. If a blog updates daily, the most it can produce is 365 entries per year, so 3MB is enough to hold one year's worth of content for 40 blog-sized websites.

If we go a bit crazy, it's reasonable to expect a few hundred megabytes of memory consumption. Let’s say the payload is 300MB—that’s 40 × 100 = 4,000 blog-sized websites for one year. This means that, without a server, and with everything distributed as static content, a user's browser client can happily serve indexed website data locally for 4,000 websites. Considering that most of the time we dwell on just a few dozen sites, and that time relevance usually plays a big role, this leaves plenty of room for imagination.

The Indexing

Call it an indexer or web crawler. In this step, we try to gather information about all available websites and process them into condensed formats like a database, CSV, or JSON.

Conceptually, the job of a web crawler could be greatly simplified if every website simply reported what it had—that’s supposedly the purpose of a sitemap. But in practice, it's surprising how much effort people spend on so-called SEO optimization, simply because there's no straightforward way to submit such information. On the other hand, perhaps it’s a good thing—we don’t want website creators to fool the search engine by submitting dishonest summary information.

One major challenge with web crawling, besides parsing and potential blocking by website hosts, is rendering JavaScript. Nowadays, lots of content is dynamic, meaning it’s not directly available in plain HTML—it may not require a server per se, but much of the visible content is rendered only during runtime. For instance, a dictionary app may only display content once a search keyword is entered.

The Searching

Database search, plain/full-text search, fuzzy search, vector search, and advanced queries—there are so many possible ways to organize data and perform searches, and so many libraries to choose from, it’s surprising that some websites and software still have such poor built-in search functions.

At the current stage, we don’t have absolute recommendations yet, but suffice it to say: advanced platforms like ElasticSearch or even a traditional database are completely unnecessary for the majority of applications. A plain full-text search is efficient enough, and lots of lightweight JavaScript or WASM libraries exist to speed up search—such as Fuse, Lunr.js, and FlexSearch. There are databases built specifically for search (not to mention the many vector databases now available for AI embedding workloads), such as Manticore Search.

The Hosting

As mentioned, the simplest search engine frontend is just a plain website that fetches index data during loading and performs search in-memory on the client side, without any server other than static file hosting. Again, it's worth stressing how powerful this can be—not only versatile and easily deployed, but also sufficient for most small to medium workloads. For typical payloads in the 3–5MB range and up to 300MB at most, this is more than enough. We can fit a lot within such constraints.

Coder Legion - Recent posts tagged search-engine

SEO Made Simple: Best Website Builders for Beginners

Why SEO is Important for Your Website

What Is An SEO Friendly Website Creator?

Picking an Easy Website Builder Online

Website Builders Comparison for Beginners

How to Create a Search Engine Friendly Website

AEO & AIO: Getting ready for Today Search

Answer Questions Directly, Use Descriptive Headlines, and Give Simple Explanations

Top Mistakes to Avoid When Building Your Website

Monitor SEO Progress Over Time

ItWeb: Web Site Building Was Never Easier

Build Smart, Get Found, Grow Online

Keep Search Clean: Block Search Engine Spam with uBlacklist and Community Rule Sets

What Problem Does uBlacklist Solve?

Step 1: How to Install uBlacklist on Chrome

Step 2: Recommended Community Rule Sets

Step 3: How to Add Recommended Rule Sets to Your Extension

Step 4: How to Verify It’s Working

How to Build a Repeatable AI Search Visibility Benchmark

How to Build a Repeatable AI Search Visibility Benchmark

What "AI search visibility" actually means (define it before you measure it)

Step 1: Build a frozen prompt corpus

Step 2: Choose your engines and cadence

Step 3: Run the manual baseline (this afternoon, free)

Step 4: Calculate your benchmark numbers

Step 5: Add context metrics the spreadsheet can't see

Step 6: Automate when manual stops scaling

The seven mistakes that break benchmarks

The bottom line

FAQ

Google Search Console Setup for Next.js (2026 Beginner's Guide)

Why This Is Tricky

Prerequisites

Step 1: Create the Google Search Console Property

Step 2: Verify Ownership

Option A — DNS TXT Record (Recommended for Domain Properties)

Option B — HTML Meta Tag (URL-Prefix Properties)

Step 3: Generate a Sitemap with sitemap.ts

Step 4: Configure robots.ts Correctly

Step 5: Submit Your Sitemap in GSC

Step 6: Verify Robots Meta Tags Aren't Blocking Individual Pages

Step 7: Use URL Inspection to Force Indexing

Step 8: Check Core Web Vitals

Step 9: Check Your Canonical URLs

Step 10: Understanding "Discovered — Currently Not Indexed"

Step 11: Don't Forget Bing Webmaster Tools

A Simple Weekly Routine

Troubleshooting Checklist

Frequently Asked Questions

Do I need a Domain property or URL-prefix property for Next.js?

How long until I see data in Google Search Console?

My sitemap shows "Couldn't fetch." What's wrong?

Can I use both a DNS TXT record and a meta tag verification?

Does adding a sitemap guarantee my pages get indexed?

Wrapping Up

Continue Learning

Free SEO Tools

I built an independent search engine—and the results were interesting

Why Updating Data in Postgres Is More Expensive Than You Think

Context

When the costs showed up

UPDATE in Postgres is not mutation

Why adding a column rewrites every row

Final design

SEO for Vibe Coders: Key Strategies to Get Your AI-Crafted Tools Discovered

Search Engine Optimisation for Vibe Coders

What Is Vibe Coding & Why SEO Needs to Adapt?

SEO Strategies Vibe Coders Should Implement

Example SEO-Christian Workflow

Why This Mindset Is Important?

Concluding Thoughts

DevLog 20250523: Sitemap and `robots.txt`

Sitemap

robots.txt

A few best practices

DevLog 20250520: Search Engine Architecture

Overview

Summary

The Indexing

Step 3: Generate a Sitemap with `sitemap.ts`

Step 4: Configure `robots.ts` Correctly