Building NLP Systems for 96 Low-Resource Languages: A Web Scraping War Story

Thu, 06 Aug 2026 16:21:00 +0000

Or why language vitality online has nothing to do with speaker counts — and everything to do with data engineering

Russia speaks dozens of minority languages — Tatar, Bashkir, Udmurt, and even tiny ones like South Yukaghir (fewer than 50 speakers). Some have no Wikipedia, no standard corpus, and almost no digital footprint.

But they do exist online — scattered across forums, blogs, and VK communities.

We wanted to measure their online presence. As NLP engineers, we quickly realized: no global corpus, no labelled dataset, no pre-trained embeddings, and no simple way to ask a search engine "give me all pages in Chechen".

So we built our own pipeline. Here's what every developer building NLP systems for low-resource languages needs to know.

The Core Problem: Low-Resource NLP Without a Dictionary

In mainstream NLP, you start with a corpus. Wikipedia. Common Crawl. Something.

For 96 minority languages, the corpus doesn't exist. You have to find the data first — before you can even think about training embeddings, fine-tuning BERT, or building a translation system.

This is the ultimate cold-start problem.

Every stage of a standard NLP pipeline breaks:

NLP Stage	The Problem
Data Acquisition	No central repository. Data is scattered across VK, forums, blogs.
Language Identification	No pre-trained classifier for 90% of these languages.
Tokenization	No standard orthography. People type without diacritics.
Embeddings	No pre-trained word vectors. You must build from scratch.
Evaluation	No gold-standard test sets. You can't measure what you can't label.

Step 1: Data Acquisition — The Search API as Your Crawler

You can't download the whole internet (unless you're Google). But search engines already crawled it.

We used Yandex.XML — a limited API with 10,000 requests per day. Each request returns one page of results, and each next page costs another request.

For an engineer, this means: you have to be extremely efficient with your queries.

The challenge: What do you search for?

We needed word markers — words that are:

Frequent in the target language
Unique — not appearing in Russian or other languages
Free of special characters — because real users don't type them

That last point is critical for any NLP system dealing with real-world user-generated content.

Most minority languages use Cyrillic extensions: ҙ, ә, ö, ҷ. There's no standard keyboard for them. People write "менан" instead of "менән". If we query with the "correct" spelling, we lose 90% of real texts.

We call this "household spelling" — like medieval birch-bark manuscripts, but with modern laziness.

For your NLP pipeline: If you're building systems for languages without standard keyboards, you need fuzzy matching at the query stage, not just at the evaluation stage. This means:

Levenshtein-based query expansion
Phonetic normalization
Handling diacritic stripping before search

Step 2: Noise Filtering — The Precision Problem

Even good markers return junk. We had to filter:

Wikipedia duplicates — we can download those in one click, so exclude them
Linguistic papers — Russian text containing one line of Rutul as an example
Music sites — song titles but no lyrics
Typos — e.g., Russian "пазух" (bosom) mistyped as "пазох", which is a valid Khakas word meaning "again"

The solution: Multi-marker verification

If one marker appears on a domain but others from the same language don't — it's probably a false positive.

For your ML pipeline: This is ensemble validation — using multiple weak signals to confirm a strong one. The trade-off is extra API calls, which are expensive at 10k/day.

Alternative approach: Train a binary classifier (Russian vs. everything else) on the fly. We did this later — more on that below.

Step 3: Domain Classification — Building a Tiered Index

We asked Yandex to categorize each domain into three tiers:

Tier	Description	Example
1. Language-first	Most pages in the minority language	Small blogs, community sites
2. Mixed	Several pages, but not dominant	Regional news portals
3. Giant platforms	Minority content is buried	YouTube, stihi.ru

Additionally, we specifically queried VK.com communities because post-2012, most standalone language sites died — and everything moved to social networks.

For your data engineering pipeline: You need a stratified crawling strategy. Don't treat all domains equally. Allocate more budget to Tier 1, and use depth-limited sampling for Tier 3.

Step 4: Language Detection Without a Training Corpus

We downloaded pages and needed to separate Russian from, say, Tuvan.

Classic n-gram language detection requires a pre-labelled corpus — which we didn't have for most languages.

The solution: We did have Russian — plenty of it. So we built a binary classifier: Russian vs. everything else.

If a paragraph isn't Russian, we assume it's our target language.

For your ML system:

This is a one-class classification problem with a twist.
It's not perfect — but it's good enough to build the first ever web corpora for these languages.
This is a classic weak supervision approach: use what you have (Russian) to learn what you don't (everything else).

What We Learned: Online vs. Offline — Zero Correlation

We compared online metrics (Wikipedia articles, VK communities, text volume) with offline metrics (speaker count, regional economy).

Metric Pair	Correlation
Among online metrics	>0.7 (Spearman)
Online vs. offline	≈ 0

This is the key insight for anyone building IT systems for low-resource languages:

How many people speak a language does not predict how alive it is on the internet.

What matters is a small, active, passionate community.

Example: Bashkir has fewer speakers than Chechen, but far more online communities. Why?

Chechnya's internet infrastructure recovered later (1990s-2000s conflicts delayed digital adoption).
Online activity is a function of infrastructure, not just population.

For your product strategy: If you're building a language model for a low-resource language, prioritize communities, not census data. Find the VK groups, Discord servers, and Telegram channels. That's where the data lives.

A Few Geeky Findings

Finding	Implication
Post-2012 shift	Almost all minority language activity moved to VK.com. Standalone forums died.
Typical user	19-31 years old, lives in the region's capital (Ufa for Bashkir, Izhevsk for Udmurt).
Typical post length	~5 words per post in minority language. Most conversation is in Russian; people sprinkle native words as identity markers.
Interconnectivity	Minority language sites link to each other across language families. A rare case of "peoples united" online.

For your NLP models: If you're building embeddings from this data, expect sparse, short texts. Standard word2vec on 5-word posts won't work well. Consider:

Character-level embeddings (more robust to spelling variation)
Cross-lingual transfer (use Russian or English as a bridge)
Zero-shot approaches (label propagation from related languages)

The Data Engineering Pipeline — Summary

┌─────────────────┐
│  Yandex.XML API │  ← 10k requests/day budget
│  (search engine)│
└────────┬────────┘
         ▼
┌─────────────────┐
│  Word Markers   │  ← Frequent + Unique + No diacritics
│  (handcrafted)  │  ← "Household spelling" handling
└────────┬────────┘
         ▼
┌─────────────────┐
│  Domain Filter  │  ← Remove Wikipedia, linguistic papers
│  (multi-marker) │  ← Ensemble validation
└────────┬────────┘
         ▼
┌─────────────────┐
│  Tiered Index   │  ← Language-first, Mixed, Giant platforms
│  (stratified)   │  ← VK.com special treatment
└────────┬────────┘
         ▼
┌─────────────────┐
│  Binary Class.  │  ← Russian vs. everything else
│  (weak super.)  │  ← No pre-labelled corpus needed
└────────┬────────┘
         ▼
┌─────────────────┐
│  Web Corpora    │  ← First ever for these languages
│  (downloadable) │  ← Available at web-corpora.net
└─────────────────┘

The Data

Downloadable corpora and domain lists:
web-corpora.net/wsgi3/minorlangs/download

And yes, there's a VK community with cute memes in minority languages — featuring cats.

The Bottom Line for Engineers

Web scraping isn't just about avoiding robots.txt. It's about understanding human behaviour — typos, keyboard limitations, social network migration, and why a language with 50 speakers can thrive online if those 50 are determined enough.

Key takeaways for building NLP systems for low-resource languages:

Start with search APIs, not crawlers. The internet is already indexed. Use it.
Handle real-world spelling. People don't use diacritics. Your queries shouldn't either.
Use weak supervision. You don't need a labelled corpus to start — use what you have.
Prioritize communities, not speakers. Online vitality ≠ offline speaker count.
Expect short texts. ~5-word posts are the norm. Plan your embeddings accordingly.

Further reading:

Our downloadable corpora: web-corpora.net
Yandex.XML API documentation
VK.com API for social media data extraction

Liked this? Share with your NLP engineering team. They'll never look at "low-resource" the same way again.

Coder Legion - Recent posts tagged language-models