Coder Legion - Recent posts tagged web-scraping

The Waiting Problem Behind Most Flaky Suites

Sat, 29 Aug 2026 21:40:45 +0000

Most of us have had a test suite that passed locally, failed once in CI, then passed on the rerun until nobody trusted it as evidence of anything. This is about why that happens, and why one framework's default behavior removes most of it.

Flake Is Usually Not A Bug

A failing browser test is rarely finding something real. It is acting on a page that has not finished arriving. The markup is there, the element is not interactive yet, and the click lands on nothing.

The usual patch is a fixed sleep. Long enough to be safe and the suite crawls, short enough to be fast and it is flaky again the first time CI has a slow day. Retries hide the problem rather than fix it, which is worse, because now a real failure looks like noise.

Auto-Waiting As A Default

Playwright makes the wait part of the action instead of a line you remembered to write. Before it clicks, it checks that the element is attached to the DOM, visible, stable and enabled. Suites converted from older tools normally lose their sleep statements outright.

It is also why the same library gets used well outside testing, for scraping, PDF generation and screenshot pipelines, where the timing problem is identical and nobody writes assertions.

No Translator In The Middle

WebDriver based tools put a separate process between your script and the browser. Playwright speaks the browser's own debugging protocol, the Chrome DevTools Protocol on Chromium, with Firefox and WebKit implementations maintained by the same team.

The commonly quoted two to five times speedup over an equivalent Selenium suite comes mostly out of that removed hop, and it widens as the suite grows. The more interesting consequence is that network interception, request mocking and multi-page control behave like first class features rather than bolt-ons.

Isolation Removes The Second Kind Of Flake

Every test runs in its own browser context, effectively a fresh incognito profile with its own cookies and storage, but cheap enough to create per test. Login state stops leaking between tests and execution order stops mattering, which kills a whole class of failure that has nothing to do with timing.

The Takeaway

Benchmark the thing that actually costs you time. Not how fast a green suite runs, but how many reruns you needed last month before you believed a result. A fuller breakdown, selectors, the Python, Java and C# bindings and the honest comparisons against Selenium and Puppeteer, is in this Playwright guide.

How Browser Agents Read A Page, And When A Script Is Still Better

Sat, 29 Aug 2026 17:44:46 +0000

Most of us here have written a scraper or a browser test suite that worked beautifully until someone shipped a redesign and every selector went stale overnight. This is a look at what actually changes when an agent reads the page instead of replaying a recorded path, and where that tradeoff stops paying for itself.

Why Recorded Steps Break

Traditional web automation targets elements by a fixed path. A class name, an xpath, a nth-child chain. That works right up until the markup moves, and then the script is not wrong in an obvious way, it is confidently clicking nothing. Anyone who has maintained a suite across a design refresh knows the failure is rarely a crash. It is a green run that tested an empty page.

An agent driving the same browser reasons about the page as it exists at that moment and finds the element by its purpose rather than its address. The login button is whatever currently behaves like the login button. That is the entire difference, and it is why the same agent can complete a task on a site it has never seen.

Three Ways An Agent Perceives A Page

Under the surface there are only a few representations in play, and most tool differences come down to how they mix them.

The DOM gives precise targeting and full text, and it drowns you in presentational noise while hiding the meaning that only layout carries. The accessibility tree, the semantic view browsers already build for screen readers, strips the clutter and labels each element by role and purpose, which usually makes it the cleanest input a model can get. And screenshots with numbered overlays on the interactive elements let a vision model say element 7 instead of describing a position in prose.

The setups that hold up in production combine them. Structure for exact targeting, the visual pass for understanding and disambiguation. A deeper breakdown of how agents control and perceive a browser is in this guide to AI browser automation, including where the frameworks and protocols underneath fit.

The Part That Still Bites

Perception does not save you from timing. Most of the modern web arrives nearly empty and fills itself in after running scripts, so an agent that reads too early acts on a page that does not exist yet. It sees a skeleton, decides confidently, and clicks a placeholder. This is the same wait-for-content problem the testing world has fought for a decade, and adding a language model does not remove it. It just makes the wrong answer more fluent.

When A Plain Script Still Wins

The honest limit is cost and determinism. An agent run costs model tokens and takes longer, and it can make a different choice on two identical pages. If a clean API exists, call it. If the target is one stable page you control, a plain script is cheaper, faster and easier to debug at 3am.

Browser agents earn their keep on the messy end: many sites with different layouts, no programmatic interface, and a task description that survives a redesign better than a selector does. That is a real category of work, and it is worth reaching for exactly there rather than everywhere.

Choosing Between A Social Platform API And Direct Collection

Sat, 22 Aug 2026 20:01:07 +0000

Most of the social data questions on here end up in the same place: someone needs posts or comments from a platform, the official API gives them two thirds of what they need, and they are trying to work out whether filling the gap is worth it. This is the decision framework I use before writing any collection code.

What The Official API Actually Gives You

An API gives you a documented schema, stable field names, and a rate limit you can plan around. That last part matters more than people expect, because a documented limit is a budget you can design a queue against instead of a mystery you discover in production.

What it does not give you is coverage. Platforms have narrowed their public endpoints repeatedly, and the fields most projects actually want, full comment trees, historical posts past a window, engagement counts on other people's content, are frequently the ones that got removed. So the API is the right default and a poor complete answer.

Where Direct Collection Fills The Gap

Direct collection reads the same public pages a logged out visitor sees. It covers the fields the API dropped, and it costs you a parsing layer that breaks whenever the markup changes.

The pattern that survives is a hybrid. Use the API wherever the endpoint exists, use direct collection only for the specific fields it cannot serve, and reconcile the two on stable ids and timestamps so you can tell which source produced a given record. If your pipeline cannot answer "where did this row come from", debugging a mismatch six months later is guesswork.

Treat A Block As Rate Feedback

Every large platform scores request patterns, browser fingerprints and timing. The useful mental model is not that you are up against a puzzle to solve, it is that you are being told your collection rate does not look like a person.

Real browser sessions, honest pacing, retries with backoff, and a hard stop when the platform signals no. Pipelines built that way keep running for months. The aggressive ones work for a week and then poison the whole project, because once you are blocked you have also lost the public access you started with.

Decide These Before You Write Code

Which specific fields do you need, and does the API already serve them. What is your collection rate, and would a human plausibly produce it. Are you touching anything behind a login, because that is the line where a public data project becomes something else. And what is your retention story, since privacy law applies to personal data even when it was posted publicly.

Answer those four and the implementation is the easy part. I wrote up the longer version, including the platform by platform data breakdown, the Python libraries, and the legal and ethical considerations, in this guide to social media scraping.

The short version: collect public data only, at a rate you would be comfortable explaining, and keep the two sources honest about where each record came from.

Building NLP Systems for 96 Low-Resource Languages: A Web Scraping War Story

Thu, 06 Aug 2026 16:21:00 +0000

Or why language vitality online has nothing to do with speaker counts — and everything to do with data engineering

Russia speaks dozens of minority languages — Tatar, Bashkir, Udmurt, and even tiny ones like South Yukaghir (fewer than 50 speakers). Some have no Wikipedia, no standard corpus, and almost no digital footprint.

But they do exist online — scattered across forums, blogs, and VK communities.

We wanted to measure their online presence. As NLP engineers, we quickly realized: no global corpus, no labelled dataset, no pre-trained embeddings, and no simple way to ask a search engine "give me all pages in Chechen".

So we built our own pipeline. Here's what every developer building NLP systems for low-resource languages needs to know.

The Core Problem: Low-Resource NLP Without a Dictionary

In mainstream NLP, you start with a corpus. Wikipedia. Common Crawl. Something.

For 96 minority languages, the corpus doesn't exist. You have to find the data first — before you can even think about training embeddings, fine-tuning BERT, or building a translation system.

This is the ultimate cold-start problem.

Every stage of a standard NLP pipeline breaks:

NLP Stage	The Problem
Data Acquisition	No central repository. Data is scattered across VK, forums, blogs.
Language Identification	No pre-trained classifier for 90% of these languages.
Tokenization	No standard orthography. People type without diacritics.
Embeddings	No pre-trained word vectors. You must build from scratch.
Evaluation	No gold-standard test sets. You can't measure what you can't label.

Step 1: Data Acquisition — The Search API as Your Crawler

You can't download the whole internet (unless you're Google). But search engines already crawled it.

We used Yandex.XML — a limited API with 10,000 requests per day. Each request returns one page of results, and each next page costs another request.

For an engineer, this means: you have to be extremely efficient with your queries.

The challenge: What do you search for?

We needed word markers — words that are:

Frequent in the target language
Unique — not appearing in Russian or other languages
Free of special characters — because real users don't type them

That last point is critical for any NLP system dealing with real-world user-generated content.

Most minority languages use Cyrillic extensions: ҙ, ә, ö, ҷ. There's no standard keyboard for them. People write "менан" instead of "менән". If we query with the "correct" spelling, we lose 90% of real texts.

We call this "household spelling" — like medieval birch-bark manuscripts, but with modern laziness.

For your NLP pipeline: If you're building systems for languages without standard keyboards, you need fuzzy matching at the query stage, not just at the evaluation stage. This means:

Levenshtein-based query expansion
Phonetic normalization
Handling diacritic stripping before search

Step 2: Noise Filtering — The Precision Problem

Even good markers return junk. We had to filter:

Wikipedia duplicates — we can download those in one click, so exclude them
Linguistic papers — Russian text containing one line of Rutul as an example
Music sites — song titles but no lyrics
Typos — e.g., Russian "пазух" (bosom) mistyped as "пазох", which is a valid Khakas word meaning "again"

The solution: Multi-marker verification

If one marker appears on a domain but others from the same language don't — it's probably a false positive.

For your ML pipeline: This is ensemble validation — using multiple weak signals to confirm a strong one. The trade-off is extra API calls, which are expensive at 10k/day.

Alternative approach: Train a binary classifier (Russian vs. everything else) on the fly. We did this later — more on that below.

Step 3: Domain Classification — Building a Tiered Index

We asked Yandex to categorize each domain into three tiers:

Tier	Description	Example
1. Language-first	Most pages in the minority language	Small blogs, community sites
2. Mixed	Several pages, but not dominant	Regional news portals
3. Giant platforms	Minority content is buried	YouTube, stihi.ru

Additionally, we specifically queried VK.com communities because post-2012, most standalone language sites died — and everything moved to social networks.

For your data engineering pipeline: You need a stratified crawling strategy. Don't treat all domains equally. Allocate more budget to Tier 1, and use depth-limited sampling for Tier 3.

Step 4: Language Detection Without a Training Corpus

We downloaded pages and needed to separate Russian from, say, Tuvan.

Classic n-gram language detection requires a pre-labelled corpus — which we didn't have for most languages.

The solution: We did have Russian — plenty of it. So we built a binary classifier: Russian vs. everything else.

If a paragraph isn't Russian, we assume it's our target language.

For your ML system:

This is a one-class classification problem with a twist.
It's not perfect — but it's good enough to build the first ever web corpora for these languages.
This is a classic weak supervision approach: use what you have (Russian) to learn what you don't (everything else).

What We Learned: Online vs. Offline — Zero Correlation

We compared online metrics (Wikipedia articles, VK communities, text volume) with offline metrics (speaker count, regional economy).

Metric Pair	Correlation
Among online metrics	>0.7 (Spearman)
Online vs. offline	≈ 0

This is the key insight for anyone building IT systems for low-resource languages:

How many people speak a language does not predict how alive it is on the internet.

What matters is a small, active, passionate community.

Example: Bashkir has fewer speakers than Chechen, but far more online communities. Why?

Chechnya's internet infrastructure recovered later (1990s-2000s conflicts delayed digital adoption).
Online activity is a function of infrastructure, not just population.

For your product strategy: If you're building a language model for a low-resource language, prioritize communities, not census data. Find the VK groups, Discord servers, and Telegram channels. That's where the data lives.

A Few Geeky Findings

Finding	Implication
Post-2012 shift	Almost all minority language activity moved to VK.com. Standalone forums died.
Typical user	19-31 years old, lives in the region's capital (Ufa for Bashkir, Izhevsk for Udmurt).
Typical post length	~5 words per post in minority language. Most conversation is in Russian; people sprinkle native words as identity markers.
Interconnectivity	Minority language sites link to each other across language families. A rare case of "peoples united" online.

For your NLP models: If you're building embeddings from this data, expect sparse, short texts. Standard word2vec on 5-word posts won't work well. Consider:

Character-level embeddings (more robust to spelling variation)
Cross-lingual transfer (use Russian or English as a bridge)
Zero-shot approaches (label propagation from related languages)

The Data Engineering Pipeline — Summary

┌─────────────────┐
│  Yandex.XML API │  ← 10k requests/day budget
│  (search engine)│
└────────┬────────┘
         ▼
┌─────────────────┐
│  Word Markers   │  ← Frequent + Unique + No diacritics
│  (handcrafted)  │  ← "Household spelling" handling
└────────┬────────┘
         ▼
┌─────────────────┐
│  Domain Filter  │  ← Remove Wikipedia, linguistic papers
│  (multi-marker) │  ← Ensemble validation
└────────┬────────┘
         ▼
┌─────────────────┐
│  Tiered Index   │  ← Language-first, Mixed, Giant platforms
│  (stratified)   │  ← VK.com special treatment
└────────┬────────┘
         ▼
┌─────────────────┐
│  Binary Class.  │  ← Russian vs. everything else
│  (weak super.)  │  ← No pre-labelled corpus needed
└────────┬────────┘
         ▼
┌─────────────────┐
│  Web Corpora    │  ← First ever for these languages
│  (downloadable) │  ← Available at web-corpora.net
└─────────────────┘

The Data

Downloadable corpora and domain lists:
web-corpora.net/wsgi3/minorlangs/download

And yes, there's a VK community with cute memes in minority languages — featuring cats.

The Bottom Line for Engineers

Web scraping isn't just about avoiding robots.txt. It's about understanding human behaviour — typos, keyboard limitations, social network migration, and why a language with 50 speakers can thrive online if those 50 are determined enough.

Key takeaways for building NLP systems for low-resource languages:

Start with search APIs, not crawlers. The internet is already indexed. Use it.
Handle real-world spelling. People don't use diacritics. Your queries shouldn't either.
Use weak supervision. You don't need a labelled corpus to start — use what you have.
Prioritize communities, not speakers. Online vitality ≠ offline speaker count.
Expect short texts. ~5-word posts are the norm. Plan your embeddings accordingly.

Further reading:

Our downloadable corpora: web-corpora.net
Yandex.XML API documentation
VK.com API for social media data extraction

Liked this? Share with your NLP engineering team. They'll never look at "low-resource" the same way again.