Coder Legion - Recent posts tagged unicode

Launching a fancy text converter: 451 styles, and the Unicode edge cases behind them

Tue, 27 Oct 2026 09:22:36 +0000

I'm Shawn Sun, a front-end engineer, and I just put a fancy text converter online. This is a developer community, so here are the parts that were fun to build instead of a feature list.

You type once and get 451 copy-and-paste styles. Tap one to copy it, and the last 8 you copied come back under Recent. No sign-up. None of these are fonts: every style is real Unicode characters, and that caused most of the work below.

1. A style is mostly a lookup table

Bold, italic, script, Fraktur and the rest live in the Mathematical Alphanumeric Symbols block, so most styles are "offset of A, plus the letter's index".

2. The block has holes

Some letters were encoded earlier elsewhere, so their slot is empty. A naive base + index gives you a box, so each style carries a small exceptions table.

3. Accents: decompose first

There is no bold é. I split it with NFD, style the e and keep the accent after it. Whether the accent then sits on top depends on the font.

4. Don't split emoji

Walk a string by code point and 👩💻 falls apart. I cut input into grapheme clusters with Intl.Segmenter and only style plain letters and digits.

5. Long input: preview first, convert on copy

Rendering every style live is fine for a username and slow for a pasted paragraph. Past 200 UTF-16 code units, each card previews only the start, and the full text is converted when you copy.

Honest limits

Conversion runs in your browser. I count anonymous usage events, such as which style was copied, but your text isn't part of them.

I'd like your feedback

Which category is missing, and do the style names make sense? Try your own text on the live site and tell me in the comments.

— Shawn Sun

How many characters your language gets in a WordPress URL

Mon, 19 Oct 2026 08:06:20 +0000

When The 200-byte trap went up, a reader on CoderLegion asked the question I should have answered in the article itself: is this only an Arabic problem?

It isn't. But "it isn't" is not much of an answer, and the one I gave in the comments was arithmetic, not measurement. So I put a typical news headline in fifteen languages through a fresh WordPress 7.1.3 site and looked at what came out the other end. Then I opened the resulting URLs the ways readers actually open them: whole, cut short, and after the slug had changed.

The short answer: English gets 200 characters. Arabic, Hebrew, Greek and Russian get about 33. Hindi, Thai, Chinese and Japanese get 22. The longer answer has a few surprises in it, and one setting that makes most of it stop mattering.

The rule: 200 characters, counted after encoding

WordPress builds a slug from the title in sanitize_title_with_dashes(). Anything outside plain ASCII is stored percent-encoded, the same %XX form it travels in, and the cut is applied to that encoded form, in one line of current core:

$title = utf8_uri_encode( $title, 200 );

Every UTF-8 byte becomes a three-character %XX token, so what a letter costs depends only on how many bytes its script needs:

a Latin letter, a digit or a hyphen: 1 byte, costs 1
Arabic, Persian, Urdu, Hebrew, Greek, Cyrillic: 2 bytes, cost 6
Hindi, Thai, Chinese, Japanese, Korean: 3 bytes, cost 9
an emoji: usually 4 bytes, costing 12, and some cost far more

Divide 200 by the cost and you have the ceiling. The cut never splits a character in half, because utf8_uri_encode() stops before one that would not fit. It does split words: nothing in it knows where a word ends. Category and tag slugs go through the same function, with the same 200.

The emoji line hides two surprises. ❤️ is two code points at 9 each, and the second is an invisible one that stays in the URL. A family emoji is three people held together by invisible joiners, and WordPress counts the joiners, then strips them, so the URL ends up with three separate people.

Every slug gets the same 200 characters. A Latin letter spends one of them, an Arabic, Hebrew, Greek or Russian letter six, a Hindi, Thai, Chinese, Japanese or Korean character nine, and an emoji twelve. What is left over is too small for one more.

The numbers, measured

Here is what WordPress 7.1.3 kept of a news headline in each language. "Cost per letter" is counted after accents are removed, so é costs what e does. "Characters kept" includes the hyphens that replaced the spaces, and the last word counted is usually a fragment.

Language	Cost per letter	Characters kept	Of which letters	Words
English	1	199	170	30
French	1	190	161	30
German	1	198	174	25
Vietnamese	1	157, the whole headline	123	35
Arabic	6	38	32	7
Persian	6	36	30	7
Urdu	6	40	32	9
Hebrew	6	38	32	7
Greek	6	37	32	6
Russian	6	36	32	5
Hindi	9	26	21	6
Korean	9	27	21	7
Thai	9	22	22	1, no spaces
Chinese	9	22	22	1, no spaces
Japanese	9	22	22	1, no spaces

Three things in that table are worth a second look.

The two-byte scripts stopped at 32 letters, not 33, and Persian at 30. Hyphens come out of the same 200, so a headline with six spaces keeps 32 letters and six hyphens, and Persian's missing two are explained further down. In practice that is five to nine words, depending on how long the words are, and the last is usually cut: the Arabic slug ends in «ال», the definite article of a word that did not fit.

Twenty-two characters is fewer letters than it sounds in Hindi and Thai. The count is of code points, and their vowel signs and tone marks are code points of their own. The Thai slug ends in «ใหม», which is «ใหม่» without its tone mark.

Thai, Chinese and Japanese get no hyphens at all. They don't put spaces between words, so the whole slug is a single run of 22 characters.

One more cost doesn't show in the table. When a slug is already taken, WordPress cuts it again to fit a suffix, so a second and a third post with the same Urdu headline each lost a letter to make room for -2 and -3. And with only five to nine words to tell headlines apart, non-Latin slugs are likelier to collide in the first place, for the same reason cut-off links go astray below.

Accented Latin gets the full 200, and the site language decides how

French, German and Vietnamese kept as much as English because their accents never reach the encoder. When a slug is saved, sanitize_title() runs remove_accents() first: é becomes e, and the whole Vietnamese headline came through as plain ASCII.

How a letter is transliterated depends on the site language. On a site set to English, «Maßnahmen» became masnahmen, and «Größe» would become grose, which is a different word. With the site set to German, the same function produces massnahmen and groesse. Danish (soester, paa), Catalan (collegi) and Serbian (djordje) get the same care from their own locales. Slugs are generated when a post is saved, so set the site language before the archive grows, not after.

What spends the budget without showing up

The cut happens before WordPress tidies the slug, so some characters are counted first and only then removed or replaced. Arabic and Persian headlines are full of them, and English ones are not free of them either.

Angle quotes. «» cost 6 each and are stripped once counted. A headline that opened with «عاجل» and quoted one more word kept one letter of «الشركات», where the same headline without the quotes kept five, «الشرك».
Invisible marks. A right-to-left mark costs 9, and so does the Persian zero-width non-joiner. Both are stripped. The Persian headline shows a second cost too: «کسبوکار» came out as «کسبوکا», glued together and then cut.
Curly punctuation, in any language. A curly apostrophe, as in what’s, costs 9 and is stripped, and so are curly quotes and the ellipsis. En and em dashes cost 9 and become hyphens. Headlines pasted from a word processor tend to be full of them: a long English headline with four apostrophes, a pair of curly quotes and a dash lost 62 characters of its slug to them, and stopped at 133.

What spends the budget and stays in the URL

Worse are the characters that are counted and then kept. WordPress's list of characters to strip was written for Latin text, so marks from other scripts reach the URL at full price.

Arabic-Indic digits. ٥٠٠٠ costs 24 where 5000 costs 4, and the cut can fall inside a number. A headline about a 5,000-dinar grant came out ending in «بقيمة-٥٠٠», worth 500: a figure that looks right and isn't. Typed with Western digits, the same headline kept the whole number.
Diacritics. Each tashkeel mark costs 6. A fully vocalised headline fitted three and a half words, «الحُكُومَةُ-تُعْلِنُ-خُطَّةً-جَدِيد».

Punctuation from other scripts is kept too, at 6 or 9 a mark. This is core's own output:

Arabic comma and question mark
عاجل،-الحكومة-تعلن-خطة؟-جديدة

Arabic semicolon
أولاً؛-ثانياً

Urdu full stop
حکومت-کا-اعلان۔-نیا-منصوبہ

Japanese punctuation
「速報」政府が新計画を発表！

Chinese punctuation
政府宣布，新计划。

The worst are invisible. WordPress strips the old bidirectional embedding marks once it has counted them, but it keeps the newer isolates, U+2066 to U+2069, and the Arabic letter mark. The isolates cost 9 each and the mark 6, and they stay in the URL without ever showing in the headline.

What happens when someone opens a cut-off link

Most sites use slug-only permalinks, /%postname%/. I opened the Arabic test post four ways:

Request	Result
the stored slug	200, the article
cut after four words, an opening two headlines share	301 to one of the two, picked by the database
cut where only one headline matches	301 to the right article
longer than the stored slug	404

The middle rows come from redirect_guess_404_permalink(). When a slug isn't found, WordPress looks for one that starts with what was requested, post_name LIKE 'cut%', and redirects to the first match. The query has no ORDER BY, so when two headlines share their opening words, which one the reader gets is up to the database. On my test site the older article won, but don't count on it: another database, or the same one reading through a different index, can return the other. News headlines share openings all the time («القبض على المتهمين في» …), and the redirect is a 301, which tells browsers and search engines the move is permanent.

If you would rather WordPress matched exactly than guessed, there is a filter for that:

add_filter( 'strict_redirect_guess_404_permalink', '__return_true' );

With it, a cut-off link gets a 404 even when the guess would have been right. Exact matches are still redirected, and that matters further down: after a switch to ID-anchored permalinks, old slug-only links are found by this same guess, and strict matching keeps them working. The filter that turns guessing off altogether, do_redirect_guess_404_permalink, would send every one of them to a 404.

The last row is the 200-byte trap in miniature. A link longer than the stored slug never matches, because the stored slug doesn't start with it. That is what readers meet when a core update shrinks a widened column under links that are already out in the world.

Put the ID in the URL

The setting that makes most of this stop mattering is under Settings → Permalinks: a custom structure of /%post_id%/%postname%/. The words stay in the URL for readers, and the number decides which article it is. The same requests, plus a few harsher ones:

Request	Result
the ID and the stored slug	200
the ID and a longer slug	301 to the article
the ID and the shared four-word cut	301 to the right article
the ID and unrelated words	301 to the article
the ID alone	200, with a canonical tag naming the full URL
an old slug-only link, after switching	301 to the article
the right slug with another post's ID	301 to that other post

The reason is a few lines in redirect_canonical(). When the slug doesn't match, the request is a 404 for a moment, but the post ID is still in it, and WordPress sends the reader to that post's real permalink:

if ( is_404() ) {
// Redirect ?page_id, ?p=, ?attachment_id= to their respective URLs.
$post_id = max( get_query_var( 'p' ), get_query_var( 'page_id' ), get_query_var( 'attachment_id' ) );

$redirect_post = $post_id ? get_post( $post_id ) : false;
// … and for a public post: $redirect_url = get_permalink( $redirect_post );

The last row in the table is the price: the number outranks the words, so a mistyped ID lands on a different article. Links are copied far more often than they are typed, and for a large archive that is a trade I would take.

Switching an existing site went smoothly in the test: old slug-only links were redirected to their articles by the same guess as above, with or without strict matching. So after switching, leave guessing on, and watch the 404 log for a while.

Give the slug its budget back

A small must-use plugin recovers the space those characters waste, and keeps Arabic-script punctuation out of the URL. It runs before WordPress counts, and only when a slug is being saved, so existing URLs keep matching exactly as before:

<?php
/**
 * Give slugs their budget back: drop or replace characters that cost slug
 * space, before WordPress counts them. Mostly Arabic, Persian and Urdu ones,
 * plus the curly quotes, dashes and ellipses that English headlines pick up.
 */
add_filter( 'sanitize_title', function ( $title, $raw_title = '', $context = 'display' ) {
if ( 'save' !== $context ) {
return $title; // Leave lookups alone, so existing URLs keep matching.
}
// Arabic diacritics, superscript alef and tatweel: 6 each, and they clutter the URL.
$title = preg_replace( '/[\x{064B}-\x{065F}\x{0670}\x{0640}]/u', '', $title );
// Quotes, the ellipsis and invisible marks, which WordPress strips anyway,
// but only after counting them.
$title = preg_replace( '/[\x{00AB}\x{00BB}\x{2039}\x{203A}\x{2018}-\x{201F}\x{2026}\x{200B}-\x{200F}\x{FEFF}]/u', '', $title );
// Direction controls: WordPress strips the embeddings only after counting
// them, and keeps the isolates and the Arabic letter mark in the URL.
$title = preg_replace( '/[\x{061C}\x{202A}-\x{202E}\x{2066}-\x{2069}]/u', '', $title );
// Punctuation that WordPress keeps in the URL (Arabic comma, semicolon and
// question mark, Urdu full stop) or turns into a hyphen only after counting
// it (no-break space, en and em dash): make it a space, which costs 1.
$title = preg_replace( '/[\x{060C}\x{061B}\x{061F}\x{06D4}\x{00A0}\x{2013}\x{2014}]/u', ' ', $title );
// Arabic-Indic and Persian digits cost 6 each; 0 to 9 cost 1.
$title = strtr( $title, array_combine(
preg_split( '//u', '٠١٢٣٤٥٦٧٨٩۰۱۲۳۴۵۶۷۸۹', -1, PREG_SPLIT_NO_EMPTY ),
str_split( '01234567890123456789' )
) );
return $title;
}, 9, 3 );

On the test site, the quoted headline's slug came out identical to the unquoted one. A long headline with an Arabic comma and question mark got plain hyphens in their place, and kept two more letters. A headline carrying invisible isolates lost them and kept three more letters, and the Arabic letter marks were dropped too. The grant headline came out with its whole number, 5000, and the vocalised headline went from three and a half words to six, with no diacritics in the URL. English titles come out as core makes them, only no longer cut short: the long English headline went from 133 characters to 195. The plugin leaves Chinese and Japanese punctuation alone, so there it still costs 9 and stays in the URL.

So, is it only Arabic?

No. Every language outside the Latin alphabet pays for the same 200 characters at a higher price: six per letter for the two-byte scripts, nine for the three-byte ones. English pays too, every time a curly apostrophe or a dash is pasted in. Arabic newsrooms are where I met it first, because that is mainly where I work, not because the rule treats Arabic differently.

What to do about it, roughly in order of effort:

If your language uses accented Latin, set the site language before you publish.
Edit long slugs by hand. Five to seven words is plenty.
On a Russian or other Cyrillic site, consider a transliteration plugin such as Cyr-To-Lat, which writes slugs in Latin letters, so they cost what English costs. Arabic rarely goes that way, because Arabic written in Latin letters is hard to read back.
On an Arabic, Persian or Urdu site, or any site whose headlines arrive with curly quotes and dashes, add the plugin above.
On a large archive, anchor permalinks on the post ID.
If you need longer slugs than any of this allows, widen the column, raise the budget, and stop core updates from shrinking it back. That is what wp-arabic-slug-schema-guard does.

Every number here was measured on a stock WordPress 7.1.3 site in WordPress Playground, which runs on SQLite. The slug lengths are worked out in PHP and will be the same on MySQL. Which article a shared opening lands on may not be, which is rather the point.

The 200-byte trap: why WordPress core updates break Arabic URLs

Tue, 22 Sep 2026 09:22:50 +0000

You update WordPress on a quiet afternoon, a routine release, the kind you've installed a hundred times. The dashboard says everything went fine. Then the 404s start: not a handful, but every long-headlined article in your archive, all at once, all in Arabic.

Nothing in the update log mentions it. No plugin changed. And the cruel part: the data that made those URLs work is already gone, shaved off inside a database-upgrade routine that ran for a few milliseconds and reported success. This isn't a freak accident or a broken plugin. It's three separate assumptions baked into WordPress core, each hard-coding the same number, 200, and Arabic sites are almost uniquely exposed to all three. We hit this running WordPress for Arabic newsrooms, the same high-traffic publishing we've written about surviving breaking-news spikes. We traced it to the exact lines in core and built a fix that survives every future update. Here's the whole story.

Why Arabic URLs hit a wall English never does

WordPress stores a post's slug in the post_name column of wp_posts, and a category or tag slug in the slug column of wp_terms. Both are VARCHAR(200) by default: room for 200 characters, which for an English headline is generous. "Everything you need to know about our new pricing" is barely fifty. You'd have to write a paragraph to run out.

Arabic is a different arithmetic, because of what WordPress actually stores in that column. It doesn't keep the raw Arabic text; it stores the percent-encoded form, the same %XX sequence that travels in the URL. Take a single word:

الذكاء  →  %d8%a7%d9%84%d8%b0%d9%83%d8%a7%d8%a1

Every Arabic letter is two bytes in UTF-8, and every byte becomes a three-character %XX token. So one Arabic character costs about six characters of column space. Do the division: VARCHAR(200) holds roughly 33 Arabic characters. A normal news headline, "القبض على المتهمين في قضية الاحتيال الإلكتروني", blows past that before it's halfway done.

So Arabic publishers learn early to widen the columns:

ALTER TABLE wp_posts MODIFY post_name VARCHAR(1024) NOT NULL DEFAULT '';
ALTER TABLE wp_terms MODIFY slug      VARCHAR(1024) NOT NULL DEFAULT '';

VARCHAR(1024) holds about 170 Arabic characters, comfortable. Permalinks resolve, the newsroom publishes, and the problem looks solved.

It isn't. It's about one-third solved, and the other two-thirds are waiting for the next core update. (This isn't only an Arabic problem, either: any non-Latin script that percent-encodes (Persian, Urdu, Hebrew, Greek) hits the same wall. Arabic publishers just hit it first; scripts that take three bytes a character, such as Hindi, Thai or Chinese, hit it harder still, at about 22 characters to Arabic's 33.)

The number 200 lives in three places, not one

Here's the insight that turns this from a recurring mystery into something you can actually close out. WordPress hard-codes "200" in three independent places, and widening the column only touches the first:

Layer	What it caps at 200	Does widening the column fix it?
1, Storage	the column that physically stores the slug	Yes, but a core update silently undoes it
2, Generation	every new slug, the moment it is created	No, it's PHP; needs a code override
3, De-duplication	slugs that collide and need a numeric suffix	No, but this only fires on collisions

Your ALTER fixes Layer 1's storage. But Layer 2 means that even with a 1024-wide column, every new slug is still cut to 200 bytes before it's ever saved, unless something on your site already overrode it. (If long Arabic slugs save correctly for you today, a plugin, your theme, or a past developer already patched Layer 2, which means you have an undocumented dependency you didn't know you were relying on.) And Layer 1, the storage you so carefully widened, is exactly the one a core update quietly reverts. Let's take each in turn.

Layer 1: how a core update erases your column width

WordPress keeps a canonical description of its own database schema, and on every major update it reconciles the live database against that description. The chain is short and deterministic:

A new release bumps $wp_db_version in version.php.
On the next admin request, wp_upgrade() notices the bump and runs the database upgrade.
That calls make_db_current_silent(), which calls dbDelta( wp_get_db_schema() ).
wp_get_db_schema() is the canonical schema, and it says, verbatim:

post_name varchar(200) NOT NULL default '',
...
slug varchar(200) NOT NULL default '',

dbDelta() then runs a DESCRIBE on every live table, compares each column to the canonical definition, and for any mismatch emits an ALTER TABLE … CHANGE COLUMN to "correct" it. Your post_name is VARCHAR(1024); the canonical says VARCHAR(200); they don't match, so dbDelta dutifully issues a CHANGE COLUMN back to 200, MySQL rebuilds the table, and every byte past the 200th is discarded.

Here's the detail that makes it unrecoverable. Years ago, core added downsize protection to dbDelta, but only for TEXT and BLOB columns (it was Trac #36748, born from exactly this kind of data loss). VARCHAR got no such guard. So a TEXT column is safe from accidental shrinking; your VARCHAR(1024) slug column is not. dbDelta shrinks it without hesitation, and restoring the column to 1024 afterwards brings back the width but not the data; those bytes were handed to MySQL and dropped.

That's why your fix keeps coming undone: always right after an update, always silently. It isn't a bug aimed at you; it's core doing exactly what it was designed to do, to a column you were never expected to change.

For accuracy: a separate routine, pre_schema_upgrade(), also runs hand-written
ALTERs, but those are version-gated to historic migrations and won't touch a
modern site's post_name. The reversion you're seeing is dbDelta.

The fix: don't repair the damage, prevent the shrink

The instinct is to write something that detects the reverted column and re-widens it after each update. Resist it. By the time you can detect the shrink, the data is already gone; you'd just be putting an empty wider column back over truncated slugs. The only fix that actually saves the data is one that stops the shrink from ever happening.

And core hands you the seam to do it. dbDelta exposes a filter, dbdelta_create_queries, that receives the canonical CREATE TABLE statements before they're compared to the live database. Rewrite the canonical post_name varchar(200) to varchar(1024) in that filter, and dbDelta's idea of "correct" now matches what's actually on disk. It sees no mismatch. It emits no CHANGE COLUMN. There is nothing to truncate.

<?php
/**
 * Plugin Name: Arabic Slug Schema Guard (must-use)
 * Description: Keeps wp_posts.post_name and wp_terms.slug at VARCHAR(1024) across
 *              core DB upgrades, so long Arabic slugs are never truncated.
 *              Prevention-first: dbDelta never even tries to shrink them.
 */
defined( 'ABSPATH' ) || exit;

const SLUG_COLUMN_LEN = 1024;

// Rewrite the canonical schema dbDelta diffs against, so "desired" already
// equals the live 1024-wide column, so no CHANGE COLUMN is ever emitted.
add_filter( 'dbdelta_create_queries', function ( $queries ) {
    global $wpdb;

    foreach ( array( $wpdb->posts => 'post_name', $wpdb->terms => 'slug' ) as $table => $column ) {
        if ( isset( $queries[ $table ] ) ) {
            $queries[ $table ] = preg_replace(
                '/(\b' . $column . '\s+varchar\()\s*200\s*(\))/i',
                '${1}' . SLUG_COLUMN_LEN . '${2}',
                $queries[ $table ]
            );
        }
    }
    return $queries;
} );

That single filter is the whole prevention story for Layer 1, and it's robust in a way an after-the-fact repair can never be: it runs inside dbDelta, so it covers every path that triggers a schema reconcile: the admin "database update required" screen, background auto-updates, and wp core update-db on the CLI alike.

Where this code lives matters as much as what it does. Put it in a must-use plugin (a single .php file in wp-content/mu-plugins/), not a regular plugin. MU-plugins always load, can't be deactivated from the dashboard, and load before the upgrade routine runs. And because a core update only ever replaces wp-admin, wp-includes and the root PHP files (it never touches wp-content), your guard is guaranteed to still be in place at the exact moment the upgrade fires. A regular plugin could be deactivated the one time it mattered; don't bet your archive on a toggle.

Layer 2: stop new slugs truncating at birth

Prevention on the column keeps your existing slugs safe. But Layer 2 is still cutting every new slug to 200 bytes at the moment of creation, inside sanitize_title_with_dashes(). The relevant line, unchanged in current core, is:

$title = utf8_uri_encode( $title, 200 );

That function is attached to the sanitize_title filter, so you can cleanly swap it for an identical copy that uses a larger byte budget:

// Layer 2: replace core's slug generator with a byte-for-byte copy whose only
// change is the encoding cap. The copy self-tests against core (see below), so a
// future core rewrite of this function is caught instead of drifting silently.
remove_filter( 'sanitize_title', 'sanitize_title_with_dashes' );
add_filter( 'sanitize_title', 'asg_sanitize_title_with_dashes', 10, 3 );

function asg_sanitize_title_with_dashes( $title, $raw_title = '', $context = 'display' ) {
    // … the body is line-for-line core, with one change:
    //     utf8_uri_encode( $title, 200 )  →  utf8_uri_encode( $title, 1000 )
    // 1000 leaves headroom under the 1024 column for a "-2" collision suffix.
}

Pick a sane budget: around 1000 bytes (≈ 160 Arabic characters) leaves room under the column for the numeric suffix WordPress adds to resolve duplicates. You don't want unlimited slugs; you want enough.

Layer 3 (_truncate_post_slug()) caps at 200 too, but it only runs when a slug collides and needs a -2 suffix. For unique news headlines that almost never fires, and it isn't filterable, so most sites can leave it alone. If you genuinely need more than 200 bytes even on collisions, you take over uniqueness through the pre_wp_unique_post_slug filter, an advanced case worth a footnote, not a paragraph.

Keeping the fork honest

Copying a core function buys a maintenance debt: the day WordPress rewrites sanitize_title_with_dashes(), your copy diverges, silently, which is the exact failure mode we're trying to kill. Pinning the copy to a core version and re-diffing by hand works right up until the release someone forgets to check, which on a long-lived site is when, not if. So don't rely on remembering. Make the plugin re-diff itself.

The lever: remove_filter() only unhooks the function; sanitize_title_with_dashes() stays callable. So you can use core's own current code as a live oracle and assert your fork still agrees with it on short inputs, where neither byte cap fires and the only thing left that could differ is the cleanup logic:

// Run once per core version. Short fixtures => neither length cap engages,
// so any difference is pure logic drift, not the byte budget.
foreach ( array( 'الذكاء الاصطناعي', 'عاجل: «مهم» اليوم', 'foo/bar — baz' ) as $f ) {
    if ( sanitize_title_with_dashes( $f, $f, 'save' )
         !== asg_sanitize_title_with_dashes( $f, $f, 'save' ) ) {
        // core changed the cleanup logic: log + email ops, and (optionally) fall
        // back to core's generator so slugs cap at 200 again: degraded, but safe
        // and known, never a malformed slug from a stale copy.
    }
}

This beats a source-hash check on two counts. It's semantic, not textual: it stays quiet when core only touches the length handling (which you override anyway) and fires only when a change actually breaks your output, so the alerts that do fire get acted on. And it degrades safely: on drift you fall back to core's exact behaviour: worst case, new slugs cap at 200 until you re-sync, never a half-encoded slug from a stale copy. Loud-and-safe beats silent-and-wrong. The full guard (with the per-core-version cache and the optional fail-safe switch) is in the complete plugin below.

A tripwire, because silent failure is the real enemy

The filter prevents the known incident. But the failure mode that actually hurt you was that it happened silently, so the last piece is making sure you'd never again learn about it from a drop in search traffic. The same MU-plugin can verify the column widths after every update and shout if anything is wrong:

Hook upgrader_process_complete to flag that a core update just ran.
On the next admin request, check the real column widths in information_schema and, if either has shrunk, write to the error log and email ops.
Expose a WP-CLI command for cron-based monitoring:

# Nightly: verify the slug columns, email ops if either has reverted.
0 3 * * *  cd /var/www/site && wp asg verify | grep -q REVERTED \
           && wp asg verify | mail -s "WP slug schema reverted on $(hostname)" *Emails are not allowed*

One discipline to internalise: this tripwire restores confidence, not data. If it ever reports a reverted column, treat it as an incident, not a self-healing event; the slugs longer than the new width are already gone and must come back from a backup. Prevention (the filter) and backups are your safety net; the tripwire just tells you when to reach for them.

"Isn't VARCHAR(1024) dangerous?": no, and the reason is reassuring

Widening these columns sounds risky (indexes, utf8mb4, InnoDB limits), and it's the part people most often get scared off by. It's almost entirely a non-issue, and one fact explains why:

The index on these columns is a fixed 191-character prefix, independent of the column's declared length. Core defines KEY post_name (post_name(191)); it indexes the first 191 characters whether the column is 200 wide or 1024. Widening the column doesn't enlarge the index by a single byte.

That dissolves the usual worries:

Index / InnoDB prefix limits. The old InnoDB prefix ceiling is 767 bytes; WordPress chose 191 because 191 × 4 (utf8mb4's worst case) = 764, just under it. Because the prefix stays at 191 no matter how wide the column grows, you never move towards that limit. (A quiet irony: slugs are stored as percent-encoded ASCII, so they were never going to hit four bytes per character anyway.)
Row size. VARCHAR stores the actual length of each value, not its declared maximum; a 30-character slug takes ~30 bytes in a VARCHAR(1024) exactly as it would in a VARCHAR(200). Rows don't get bigger because the ceiling moved up.
Query plans. Lookups use the same 191-prefix index to narrow, then verify the full value. Identical behaviour at 200 or 1024.
Plugins. SEO plugins, caches and CDNs read post_name and emit longer URLs without complaint.

The genuine risks aren't in steady state; they're tools that recreate the schema from an old definition: a migration plugin that runs its own CREATE TABLE, or, most commonly, restoring a SQL dump taken before you widened the columns. A dump of the current database preserves VARCHAR(1024) correctly; importing an old structure silently puts you back at 200. Write that one down in your runbook.

Doing it safely on a site with millions of posts

There's one operational gotcha, and it's about the migration itself, not the result. Going from VARCHAR(200) to VARCHAR(1024) cannot be an instant, in-place change. InnoDB only keeps a VARCHAR length change "in place" while both lengths stay in the same size class: values under 256 use one length byte, values 256 and over use two. Crossing 255 forces ALGORITHM=COPY: a full table rebuild. On a wp_posts with millions of rows, a raw ALTER means a long operation and a problematic lock.

So on a large live site, do the initial widening with an online schema-change tool, not a bare ALTER:

pt-online-schema-change \
  --alter "MODIFY post_name VARCHAR(1024) NOT NULL DEFAULT ''" \
  --chunk-time=0.5 --max-load Threads_running=50 \
  --execute D=wordpress,t=wp_posts

(gh-ost is an equally good triggerless alternative.) wp_terms is tiny and takes a plain ALTER without ceremony; wp_posts is the one that needs the online tool. The rollout that's bitten no one:

Back up, and confirm the dump shows VARCHAR(1024) afterwards.
Clone to staging, deploy the MU-plugin, and run a real major core update there. Confirm no CHANGE COLUMN appears and the verify command reports OK.
Deploy the MU-plugin to production before the next core update: prevention has to be in place ahead of the event it prevents.
Widen the production columns off-peak with pt-osc / gh-ost.
Add a nightly verify cron and a 404 monitor, so anything unexpected surfaces in hours, not in a month of lost rankings.
Keep the MU-plugin in version control, and never import a pre-widening structure dump.

The deeper fix: stop betting your URLs on a column width

Everything above makes Arabic slugs safe. But there's a more permanent way to think about it, and for a large news archive it's worth the shift: don't let the URL depend on the slug surviving at all.

Anchor your permalinks on the numeric post ID, and keep the Arabic slug as a descriptive (but cosmetic) suffix:

https://example.com/123456/الذكاء-الاصطناعي

Resolve the post by its ID, and the slug becomes decoration. If it's ever truncated, malformed, or edited, the URL still resolves to the right article instead of 404-ing, and WordPress can 301 to the canonical form. You keep the Arabic keywords in the URL for search relevance and you become structurally immune to the entire class of problem. (If you want to stop touching core tables altogether, you can go one step further and store the long slug in your own indexed routing table, which dbDelta has no opinion about, but for most teams, ID-anchored permalinks are the sweet spot.)

That's the difference between patching a bug and designing it out.

The complete plugin

Everything above, assembled into one must-use plugin. Drop it in wp-content/mu-plugins/wp-arabic-slug-schema-guard.php, widen the columns once with an online schema-change tool, and all three layers, plus the tripwire, are handled in a single place. The slug generator is a copy of core's sanitize_title_with_dashes(); only the byte cap differs.

📦 Prefer it ready-made? The plugin is open-source on GitHub as mantekio/wp-arabic-slug-schema-guard, or composer require mantekio/wp-arabic-slug-schema-guard. Star it, file an issue, or lift it whole.

<?php
/**
 * Plugin Name: Arabic Slug Schema Guard (must-use)
 * Description: Keeps wp_posts.post_name and wp_terms.slug at VARCHAR(1024) across
 *              core DB upgrades, lets core generate long URL-encoded Arabic slugs,
 *              and verifies + alerts after every update. Prevention-first:
 *              dbDelta never truncates, because the canonical schema it diffs
 *              against already says 1024. The slug fork self-tests against core
 *              on each update and can fail safe.
 */
defined( 'ABSPATH' ) || exit;

const ASG_COLUMN_LEN = 1024;  // physical column width (bytes)
const ASG_SLUG_BYTES = 1000;  // max generated slug length, under the column width
const ASG_SCHEMA_SIG = 'post_name:1024;slug:1024';
// define( 'ASG_ALERT_EMAIL', '*Emails are not allowed*' );  // optional, set in wp-config.php
// define( 'ASG_L2_FAILSAFE', true );               // optional: on Layer-2 drift, fall back to core (slugs cap at 200) until you re-sync

/* LAYER 1: PREVENTION
 * Rewrite the canonical CREATE TABLE dbDelta diffs against, so "desired" already
 * equals the live 1024-wide column and no destructive CHANGE COLUMN is emitted. */
add_filter( 'dbdelta_create_queries', function ( $queries ) {
    global $wpdb;
    foreach ( array( $wpdb->posts => 'post_name', $wpdb->terms => 'slug' ) as $table => $column ) {
        if ( isset( $queries[ $table ] ) ) {
            $queries[ $table ] = preg_replace(
                '/(\b' . $column . '\s+varchar\()\s*200\s*(\))/i',
                '${1}' . ASG_COLUMN_LEN . '${2}',
                $queries[ $table ]
            );
        }
    }
    return $queries;
} );

/* LAYER 2: GENERATION
 * Core caps new slugs at 200 bytes via utf8_uri_encode($title, 200). Swap in a
 * copy of sanitize_title_with_dashes() whose only change is the byte budget.
 * Installed conditionally: if the drift guard (below) flagged THIS core version
 * and ASG_L2_FAILSAFE is on, stay on core's default (caps at 200, degraded but
 * safe and known) rather than run a fork we know has drifted. */
if ( get_option( 'asg_l2_status' ) === 'drift:' . $GLOBALS['wp_version']
     && defined( 'ASG_L2_FAILSAFE' ) && ASG_L2_FAILSAFE ) {
    // Known-drifted on this core version → leave core's generator in place.
} else {
    remove_filter( 'sanitize_title', 'sanitize_title_with_dashes' );
    add_filter( 'sanitize_title', 'asg_sanitize_title_with_dashes', 10, 3 );
}

function asg_sanitize_title_with_dashes( $title, $raw_title = '', $context = 'display' ) {
    $title = strip_tags( $title );
    // Preserve already-encoded octets through the cleanup below.
    $title = preg_replace( '|%([a-fA-F0-9][a-fA-F0-9])|', '---$1---', $title );
    $title = str_replace( '%', '', $title );
    $title = preg_replace( '|---([a-fA-F0-9][a-fA-F0-9])---|', '%$1', $title );

    if ( seems_utf8( $title ) ) {
        if ( function_exists( 'mb_strtolower' ) ) {
            $title = mb_strtolower( $title, 'UTF-8' );
        }
        $title = utf8_uri_encode( $title, ASG_SLUG_BYTES );  // core hard-codes 200 here
    }

    $title = strtolower( $title );

    if ( 'save' === $context ) {
        // EVERYTHING here is copied verbatim from core. The ONLY sanctioned difference
        // between this fork and core is the byte budget above. Drop a line and you have
        // drifted, see the note under the drift guard below.

        // Turn non-breaking spaces, en/em dashes and slashes into hyphens.
        $title = str_replace( array( '%c2%a0', '%e2%80%93', '%e2%80%94' ), '-', $title );
        $title = str_replace( array( ' ', ' ', '–', '–', '—', '—' ), '-', $title );
        $title = str_replace( '/', '-', $title );

        // Strip these characters entirely. Note %c2%ab / %c2%bb: the guillemets Arabic
        // uses as quotation marks. And %e2%80%8e / %e2%80%8f: the LRM and RTL marks.
        $title = str_replace(
            array(
                '%c2%ad',                                            // soft hyphen
                '%c2%a1', '%c2%bf',                                  // iexcl, iquest
                '%c2%ab', '%c2%bb', '%e2%80%b9', '%e2%80%ba',        // angle quotes
                '%e2%80%98', '%e2%80%99', '%e2%80%9c', '%e2%80%9d',  // curly quotes
                '%e2%80%9a', '%e2%80%9b', '%e2%80%9e', '%e2%80%9f',
                '%e2%80%a2',                                         // bullet
                '%c2%a9', '%c2%ae', '%c2%b0', '%e2%80%a6', '%e2%84%a2',
                '%c2%b4', '%cb%8a', '%cc%81', '%cd%81',              // acute accents
                '%cc%80', '%cc%84', '%cc%8c',                        // grave, macron, caron
                '%e2%80%8b', '%e2%80%8c', '%e2%80%8d',               // zero-width chars
                '%e2%80%8e', '%e2%80%8f',                            // LRM, RTL mark
                '%e2%80%aa', '%e2%80%ab', '%e2%80%ac',
                '%e2%80%ad', '%e2%80%ae',
                '%ef%bb%bf',                                         // byte order mark
                '%ef%bf%bc',                                         // object replacement
            ),
            '',
            $title
        );

        // Convert × to 'x'.
        $title = str_replace( '%c3%97', 'x', $title );
    }

    $title = preg_replace( '/&.+?;/', '', $title );          // strip remaining entities
    $title = str_replace( '.', '-', $title );
    $title = preg_replace( '/[^%a-z0-9 _-]/', '', $title );  // keep only slug-safe chars
    $title = preg_replace( '/\s+/', '-', $title );
    $title = preg_replace( '|-+|', '-', $title );
    return trim( $title, '-' );
}

/* LAYER 2 DRIFT GUARD: self-test the fork against core's live implementation.
 * remove_filter() only unhooks sanitize_title_with_dashes(); the function stays
 * callable, so we use core's CURRENT code as a live oracle and assert our fork
 * still agrees on SHORT inputs (where neither byte cap engages, the only thing
 * the fork changed). Any divergence == core altered the cleanup logic and our
 * copy has drifted. Runs once per core version; shouts, and can fail safe. */
add_action( 'admin_init', 'asg_guard_layer2_drift' );

function asg_guard_layer2_drift() {
    $ok      = 'ok:' . $GLOBALS['wp_version'];
    $drifted = 'drift:' . $GLOBALS['wp_version'];
    if ( in_array( get_option( 'asg_l2_status' ), array( $ok, $drifted ), true ) ) {
        return;  // already settled for this core version
    }
    if ( ! function_exists( 'sanitize_title_with_dashes' ) ) {
        return;  // core refactored it away: no oracle to diff against; the fork
                 // still runs as a standalone copy (it needs no removed function).
    }

    // Deliberately SHORT fixtures so neither length cap fires; any difference is
    // pure cleanup-logic drift, not the byte budget.
    $fixtures = array(
        'الذكاء الاصطناعي',
        'عاجل: تطورات «مهمة» اليوم',
        'Mixed عربي + English — dash',
        'foo/bar — baz',
        'UPPER   multiple   spaces',
        '%d8%a7 pre-encoded octet',
    );

    $drift = array();
    foreach ( $fixtures as $f ) {
        $core = sanitize_title_with_dashes( $f, $f, 'save' );
        $ours = asg_sanitize_title_with_dashes( $f, $f, 'save' );
        if ( $core !== $ours ) {
            $drift[] = "in=[{$f}] core=[{$core}] ours=[{$ours}]";
        }
    }

    if ( $drift ) {
        $msg = "[Arabic Slug Schema Guard] Layer-2 DRIFT on WP {$GLOBALS['wp_version']}: core "
             . "sanitize_title_with_dashes() no longer matches our fork; re-sync the copy.\n"
             . implode( "\n", $drift );
        error_log( $msg );
        if ( defined( 'ASG_ALERT_EMAIL' ) && function_exists( 'wp_mail' ) ) {
            wp_mail( ASG_ALERT_EMAIL, 'WP slug Layer-2 drift: ' . wp_parse_url( home_url(), PHP_URL_HOST ), $msg );
        }
        update_option( 'asg_l2_status', $drifted, true );  // autoload: the fail-safe reads it every request
        return;  // do NOT cache "ok" while drifted
    }
    update_option( 'asg_l2_status', $ok, true );
}

/* LAYER 3: DE-DUPLICATION (optional)
 * _truncate_post_slug() also caps at 200, but only when a slug COLLIDES and needs
 * a "-2" suffix, rare for unique headlines, and it isn't filterable. If you need
 * >200 bytes even on collisions, take over uniqueness here. Most sites skip this.
 *
 * add_filter( 'pre_wp_unique_post_slug', 'asg_unique_post_slug', 10, 6 );
 */

/* TRIPWIRE: verify + alert after every core update
 * This only detects and alerts: it restores the COLUMN, never bytes already
 * truncated. Treat any revert as an incident: restore from backup. */
add_action( 'upgrader_process_complete', function ( $upgrader, $extra ) {
    if ( isset( $extra['type'] ) && 'core' === $extra['type'] ) {
        delete_option( 'asg_schema_sig' );  // force a re-check on the next admin_init
    }
}, 10, 2 );

add_action( 'admin_init', function () {
    if ( get_option( 'asg_schema_sig' ) === ASG_SCHEMA_SIG ) {
        return;  // fast path: already verified since the last update
    }
    $reverted = asg_reverted_columns();
    if ( $reverted ) {
        $msg = '[Arabic Slug Schema Guard] Column(s) reverted: ' . implode( ', ', $reverted )
             . '. Long slugs were truncated: restore from backup and check 404 logs.';
        error_log( $msg );
        if ( defined( 'ASG_ALERT_EMAIL' ) ) {
            wp_mail( ASG_ALERT_EMAIL, 'WP slug schema reverted: ' . wp_parse_url( home_url(), PHP_URL_HOST ), $msg );
        }
        return;  // do NOT cache "ok" while broken
    }
    update_option( 'asg_schema_sig', ASG_SCHEMA_SIG, false );
} );

function asg_reverted_columns() {
    global $wpdb;
    $reverted = array();
    foreach ( array( array( $wpdb->posts, 'post_name' ), array( $wpdb->terms, 'slug' ) ) as list( $table, $column ) ) {
        $len = (int) $wpdb->get_var( $wpdb->prepare(
            "SELECT CHARACTER_MAXIMUM_LENGTH FROM information_schema.COLUMNS
              WHERE TABLE_SCHEMA = %s AND TABLE_NAME = %s AND COLUMN_NAME = %s",
            DB_NAME, $table, $column
        ) );
        if ( $len && $len < ASG_COLUMN_LEN ) {
            $reverted[] = "{$table}.{$column} (now {$len})";
        }
    }
    return $reverted;
}

/* WP-CLI: `wp asg verify` for cron-based monitoring */
if ( defined( 'WP_CLI' ) && WP_CLI ) {
    WP_CLI::add_command( 'asg verify', function () {
        foreach ( asg_reverted_columns() ?: array( 'ok' ) as $row ) {
            WP_CLI::log( 'ok' === $row ? 'Schema OK: both columns at VARCHAR(1024).' : "REVERTED: {$row}" );
        }
    } );
}

A postscript: the drift guard caught us

The argument above is that a fork of a core function drifts silently, so the guard matters more than the copy. That turned out to be true in the least comfortable way.

Versions 1.0.0 through 1.2.0 of the released plugin shipped a fork that was missing core's entire "strip these characters entirely" block, along with the × to x mapping. The consequence lands exactly on the readers this post is written for: Arabic guillemets (« ») and the LRM/RTL direction marks survived into slugs as %c2%ab, %c2%bb, %e2%80%8f. The character filter at the end cannot remove them, because %c2%ab is built entirely from permitted characters.

The drift guard printed above would have caught it on the first admin_init. Its second fixture is 'عاجل: تطورات «مهمة» اليوم'. It never fired, because the guard is bound to admin_init and nobody had opened wp-admin on an install running that fork. A guard on a hook that never fires is a comment.

Both are fixed in v1.3.0: the fork is re-synced (the block above is now complete and verified against core by an offline oracle on every build), and wp asg verify evaluates the drift oracle itself rather than waiting for an admin page load. If you lifted this code before July 2026, take the str_replace block above.

This is the kind of problem that never shows up in a tutorial. It lives where WordPress internals, MySQL's storage rules, Arabic character encoding and SEO all overlap, and it only reveals itself at the worst possible moment. It's also the territory we work in every day: keeping Arabic publishing fast, multilingual and unbreakable on WordPress, right through to AWS at scale.

If you run a WordPress publication in Arabic and want it to survive its own updates, let's talk.

I hid an entire webpage inside a cat face

Fri, 22 May 2026 10:33:59 +0000

The source of this page is just a cat face:

https://nothing-to-see-here.surge.sh/

View source. You'll see (=･ω･=) and nothing else.
But the page runs. Rainbow animation, layout, everything.

The entire JavaScript is encoded as invisible Unicode
Variation Selectors attached to the cat emoji.

Unicode VS (U+FE00–FE0F and U+E0100–E01EF) map
precisely to 256 byte values. Any byte sequence can
ride inside normal text, invisible to readers,
surviving copy-paste across Slack, X, LINE, iMessage.

That's subtext.

https://prospectorlabs.dev/subtext

Unicode escape' codec can't decode bytes in position 2-3: truncated \ uxxxxxxxx escape

Sat, 09 May 2026 07:21:58 +0000

In this article, we will solve a syntax based problem of "Unicodeescape' codec can't decode bytes in position 2-3: truncated \uxxxxxxxx escape". This problem is encountered when we make a mistake in using the "unicode esacpe characters."Python's interpreter is designed to understand certain characters with certain symbols and in programming each character is given a unicode. This error occurs when the interpreter fails to distinguish between a normal character and a unicode character. We can solve this problem by simply manipulating the string according to the interpreter's logic. We will also discuss how these unicode character works and the reason behind this syntax problem. Let's quickly go through the contents of this article.

Table of contents: -

Understanding the error and discussing how the interpreter functions.
Solving the error with a relevant example
Why unicode characters were assigned and steps to avoid this error.
The Conclusion

Understanding the error and discussing how the interpreter functions. #

Let's break this error statment down: "Unicodeescape' codec can't decode bytes in position 2-3: truncated \uxxxxxxxx escape". This means that there is a syntax error and the interpreter is not able understand the meaning of this code. The "Unicodeescape" term means that the syntax error is caused because the unicode escape characters are not used properly. Now we have a lot to dissect, let's start with the interpreter's logic. if we pass a command like: -


print('This is her mother's dress')

The output of this code will be: -


print('This is her mother's dress.')
          ^^^^^^^^^^^^^^^^^^^^^
SyntaxError: invalid syntax. Perhaps you forgot a comma?

Here, the interpreter is unable to understand the range of the string. We use " ' " character to set the range of characters to be printed. The interpreter is confused because it is not able to distinguish between a normal character and a single quote range setter. We can solve this problem with the help of escape sequences.


print('This is her mother\'s dress')


This is her mother's dress

Here , the backslash character instructs the interpreter that it is an escape sequence for the single quote character ('). As we observed that the interpreter was confused and this confusion led to a syntax error. This is the exact problem we are facing. Let's understand this with the help of an example.

Solving the error with a relevant example#

We know that the interpreter follow certain rules while printing and traversing the data. Suppose we have to load a file and traverse the data stored in it. We use the file handling concepts and perform the operation but on running the program we encounter the same error: -


path = "C:\Users\Dell\Downloads\lamdaphagevirus\lamda.txt"
file = open(path, "r")
data=file.read()
print(data)


path = "C:\Users\Dell\Downloads\lamdaphagevirus\lamda.txt"
                                                              ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape

What went wrong here? The error is in the path variable. we passed the location of the file and it consists of multiple backslashes. Now the interpreter is confused which character is a normal character and which one is a unicode escape character. We know that the entire file location is a normal string without any unicode characters but the use of "\" characters is causing the problem. How are we going to solve this problem? Let's look at some solutions: -

Tip Each character is assigned with a unicode and an escape sequence which is universal for all the languages.

Solution 1

For the 1st solution, we will convert out string into a raw string like this: -


path = r"C:\Users\Dell\Downloads\lamdaphagevirus\lamda.txt"
file = open(path, "r")
data=file.read()
print(data)


GGGCGGCGACCTCGCGGGTTTTCGCTATTTATGAAAATTTTCCGGTTTAAGGCGTTTCCGTTCTTCTTCG
TCATAACTTAATGTTTTTATTTAAAATACCCTCTGAAAAGAAAGGAAACGACAGGTGCTGAAAGCGAGGC

A raw string is used whenever we want to avoid the technique of escaping characters with the help of backslashes. In order to convert a string into a raw string we have to simply add a "r" before the starting of the string. If this solution does not work for you then try the next method.

After this manually install "dlib" from this link. This can help the programmer to solve this issue.

Solution 2

If the the raw string method fails then try to escape the characters with the help of a backslash.


path = "C:\\Users\\Dell\\Downloads\\lamdaphagevirus\\lamda.txt"
file = open(path, "r")
data=file.read()
print(data)


GGGCGGCGACCTCGCGGGTTTTCGCTATTTATGAAAATTTTCCGGTTTAAGGCGTTTCCGTTCTTCTTCG
TCATAACTTAATGTTTTTATTTAAAATACCCTCTGAAAAGAAAGGAAACGACAGGTGCTGAAAGCGAGGC

For the interpreter a single backslash character means this "\\". All the escape characters begin with a backslash and by using this character before every backslash in the string, we can instruct the interpreter to treat this character as a escape sequence.

Solution 3

For the third solution we have to manipulate the string and replace every backslash with a "/" character.


path = "C:/Users/Dell/Downloads/lamdaphagevirus/lamda.txt"
file = open(path, "r")
data=file.read()
print(data)


GGGCGGCGACCTCGCGGGTTTTCGCTATTTATGAAAATTTTCCGGTTTAAGGCGTTTCCGTTCTTCTTCG
TCATAACTTAATGTTTTTATTTAAAATACCCTCTGAAAAGAAAGGAAACGACAGGTGCTGAAAGCGAGGC

Note In case of file handling, majority of the errors are caused due to an invalid location entry.

To know more about unicode errors vist this link.

Why unicode characters were assigned and how to avoid this error.#

Now that we have discussed the different solutions for the unicode error, we will try to understand the logic behind the designation of such codes. The main reason for assigning a universal code to every character was to establish uniformity in programming. During the early phases when technology was developing, majority of the codes were built around English language but as the technology progressed many nations started to get involved in this field. So, in order to avoid confusion and establish uniformity among the programming world, codes were assigned which were universal for all the languages around the globe.

We have to learn more about escape sequences and unicode if we want to avoid this kind of syntax error.

The conclusion

This article offers the most effective solution for the error: - "Unicodeescape' codec can't decode bytes in position 2-3: truncated \uxxxxxxxx escape". We discussed about the logic of interpreter and the different ways to solve this error. We strolled through the concepts of unicode and escape sequences. We also discussed the idea and purpose behind the creation of unicode.

Unicodedecodeerror: 'utf-8' codec can't decode bytes in position 0-1: invalid continuation byte

Fri, 27 Mar 2026 02:21:43 +0000

When working with strings you may have encountered the error "UnicodeDecodeError: 'utf-8' codec can't decode byte X in position X: invalid continuation byte". It occurs because when we specify an incorrect encoding when decoding bytes data. In order to fix the issue, we have to specify correct encoding.

Table of Contents: #

UnicodeDecodeError: 'utf-8' codec can't decode byte X in position X: invalid continuation byte #

Managing the encoding of character strings can sometimes cause problems in Python. For example, the error "UnicodeDecodeError: 'utf-8' codec can't decode byte xxx in position x: invalid continuation byte" occurs in a script in Python when trying to decode a string to UTF-8 but that it is not encoded in this way. If you are handling a file and you don't know the encoding, there are solutions to work around this error.

To understand what the errors means, we have to analyze the error message. Strings have encoding that describes character set and collation the string bytes objects will accept. When we encode the string in UTF-8, we specify data in UTF-8 standard format. If we were to decode the string, we were to decode the string with latin-1 for instance, that will cause the error "UnicodeDecodeError: 'utf-8' codec can't decode byte in position: invalid continuation byte".

Encoding is a process whereby we turn sequence of characters, which includes alphabet, numbers, punctuation and all of the other symbols, into bytes for efficiency in transmission and storing. Decoding is the opposite of encoding. It's a process of turning bytes into the sequence of characters.

Here's a demonstration of the problem:

str_bytes = 'ééééééé'.encode('latin-1')

# ⛔️ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 0: invalid continuation byte
my_str = str_bytes.decode('utf-8')

The code runs into an error when we try to decode the string with an encoding that wasn't used to decode the string. To fix this we have to decode the string using the same encoding we used to encode it. Here's the fix to the previous error:

str_bytes = 'ééééééé'.encode('latin-1')
my_str = str_bytes.decode('latin-1')

Now the program runs correctly and no errors occur.

The Solution #

In general, you ought to decode your strings in the same encoding you used to decode but at times this issue is hard to avoid because it can be difficult to keep track of many encoding. That's why I present two solution. The first one is to decode your strings correctly and the second one is to read the strings from a file and have Python automatically find the string encoding.

Solution one

When doing a decoding of bytes using decode() method, you have to make sure that the encoding is the one was used to encode the string into bytes. For example, in this example I have set the encoding of first_string to UTF-8 which is an encoding that accepts all known language on earth, and then to decode it I have set the encoding to UTF-8 again.

first_string = 'ééééééé'.encode('utf-8')
decoded_str = first_string.decode('utf-8')
print(decoded_str)

Solution two

File management functions have a "Binary" mode which treats characters as "bytes". With this mode, no decoding is performed and the characters are thus preserved, whatever their encoding. To open a file in binary mode, you must specify the "rb" mode.

with open(filePath, 'rb') as file:
    content = file.read()

To write to a file in binary mode, you must use the "wb" or "ab" modes.

If, despite the encoding problem, you want to open the file and read the content in utf-8, it is possible to add an additional parameter to the "open()" function telling it to ignore errors. Characters that cannot be read will be ignored and not displayed.

with open(filePath, encoding="utf8", errors='ignore') as file:

The "byte 0xff in position 0" error that appears when you try to decode a file in UTF-8 may simply indicate that the file is encoded in UTF-16. You can try changing the opening encoding of the file.

with open(filePath, encoding='utf-16') as file:

This solution only works with Python 3, which includes UTF-16 encoding support in the "open()" function. If you are using Python 2, you will have to perform a conversion after opening the file in binary mode.

with open(filePath,'rb') as file:
    content = file.read()
content = content.rstrip("\n").decode("utf-16")

The Conclusion

Thank you for sticking with this tutorial all the way to the end. When attempting to decode a string using decode() method and passing wrong encoding as a parameter the error message "UnicodeDecodeError: 'utf-8' codec can't decode byte X in position X: invalid continuation byte". To fix this issue, you have to either find out what encoding was used for the string in encode() method and use it to decode the string using decode() method. If you are reading encoding from a file and not a string, you can open the file in binary mode with open() method and pass the correct encoding to it's encoding parameter.

If you found this article helpful. Please comment and share. If you find any issue you can always tell, I'll get back to you as soon as possible. Take care!

Data Types in Java, Part 1, the primitive types in detail

Mon, 15 Sep 2025 20:00:25 +0000

Data Types in Java

All the data types in Java have been categorized into two, primitive data types
and the reference data types.

The primitive types, also known as built-in types, are byte, short, int, long,
float, double, char and boolean.

The reference data types are arrays, classes, interfaces, enums, annotations
and records.

enums and annotations were introduced into the Java programming
language from Java 5 and records were introduced into the Java Programming
Language from Java 16.

From Java 8, there is another type called Functional Interface, which is identified by compiler. The Functional Interface is a kind of interface, for which instances may be created using lambda expressions(introduced into the Java Programming Language from java 8).

Primitive Data Types

The following table list the primitive types in Java:

The primitive types may be further categorized into numeric types and boolean
type.

boolean Data Type

A variable of the boolean type can have only two possible values, true or false. In Java, boolean is a separate data type and is not like C language, where a numeric value of non-zero or zero is considered as true or false, i.e. a numeric type cannot be used as boolean. E.g. we cannot use a numeric expression in the condition of the if statement, as is allowed in C, where boolean is not a separate data type. There are many operators on other data types which result in value of type boolean, and all the Logical operators available for boolean
type also result in value of boolean. The various operators will been discussed in a separate tutorial.

Numeric Data Types

In the Java programming language, we do not have the signed or the unsigned prefix for the numeric data types. Note that signed and unsigned are not keywords in Java.

Among the numeric types, byte, short, int and long are the signed integral types, char is an unsigned integral type and the float and double are single-precision and double-precision signed floating-point data types.

byte order

For all the numeric types whose size is more than 1 byte, there are two ways of ordering the bytes. The byte order for a machine could be big-endian or little-endian. The byte ordering is normally dependent on the architecture of the machine. All of the ×86 and ×64 machines follow the little-endian byte ordering for the numeric types, and most of the other machines, with RISC-based architecture follow the big-endian byte ordering. The Java Virtual Machine
follows the big-endian byte ordering. The difference in the two types of byte orderings is that in case of the little-endian byte ordering, the most significant byte is at a higher location (comes at the end) and the least significant byte is at a lower location (comes first), whereas in case of the big-endian byte ordering, the most significant byte is at a lower location (comes first) and the least significant byte is at a higher location (comes at the end). The following figure shows the difference in byte ordering.

integral data types

What is the size of int in C? Is it 2 bytes or 4 bytes? It depends on the platform. Most of the C compilers on the ×86 platform consider int as 2 bytes, whereas on the RISC machines, it is commonly found to be using 4 bytes, in some cases it may even use 8 bytes. Java works on single platform, the JVM; so on this platform the sizes of each of the data types are fixed. The size of byte is 1 byte, short is 2 bytes, int is 4 bytes and long is 8 bytes. The integral values are stored using 2’s complement. The range of values for the various integral data
types is given in Table below:

floating-point data types

The types float and double represent the single-precision and double-precision floating-point values and their sizes are 4 and 8 bytes, respectively. These floating-point data type representations are according to the IEEE-754 standard for single-precision (32-bit) and double-precision (64-bit) floating-point values. This standard uses the most significant bit (MSB) as the sign bit, and the rest of the bits are divided among two fields, an exponent and a significand. In case of the single-precision value (float of Java), bit-31 is the sign bit, bits 30–23 are used for the exponent and bits 22–0 are used for the significand. In case of the double-precision value (double of Java), bit-63 is the sign bit, bits 62–52 are used for the exponent and bits 51–0 are used for the significand. The range of values for the two floating-point data types is given in Table below:

Infinities and NaNs for floating-point types

Let us consider the code given below:

public static void main(String[] args) {
    double a = 75;
    double b = 0;
    double c = a / b;
    System.out.println(c);
}

What would be the output of executing the above code? Error at line 4 (divide y zero)? No this does not give any error at runtime and executes perfectly. What, is the value of dividing 75 by 0? The above code prints Infinity. If we use int instead of the double data type in the above code then we will get an error at runtime, but for the floating point data types, division by zero is not an error at Runtime. The floating-point data types can represent infinities (positive and negative) as well as not-a-number (result of 0 divide by 0).

According to the IEEE 754, the floating-point numbers have a representation for positive infinity and negative infinity. There are also representations for the values that are Not-a-Number (NaN).

When all bits for the exponent field are one, then these numbers are used to represent the infinities and the NaNs. The figures below shows the bit representations for the infinities and the NaNs.

When all exponent bits are one and all significand bits are zero, they represent infinity. Depending on the sign bit, the infinity is either a positive infinity or a negative infinity. When all exponent bits are one and the significand is a non-zero value, then they represent NaN.

floating-point types and precision

The use of floating-point data types float and double is discouraged, since these data types have a limitation about precision level. These types cannot precisely represent all the floating-point values. The precision level falls significantly as magnitude of value increases (moving away from zero). for e.g. if, we consider the float data type, then we find that float is incapable of represnting 123456789. The float can represent the value 123456784 and then 123456792. We can try the code in Listing below to test this.

class TestFloatingPoint {
    public static void main(String[] args) {
        for (int i = 123456784; i < 123456793; i++) {
            float f = i;
            System.out.printf(”int:%d, float:%10.1f\n” , i, f ) ;
        }
    }
}

Atleast, when dealing with larger values do not use the and float and double data types. When dealing with very high values, we have the option of using the classes like java.math.BigInteger and java.math.BigDecimal, we will discuss them in another tutorial.

char data type

char data type in a programming language is used to represent a unit of text. How is text data represented? The text data is represented as sequence of characters. The char data type is simply a numeric value of the character from a character set.

Character sets.

What is a character set? A character set is a collection of a unit of text(character) which are assigned some unique numeric value. There are various character sets available. The most commonly known character set is the ASCII(American Standard Code for Information Interchange) character set which assigns only 128 characters(including the control characters) to the numeric values in the range from 0 — 127. Not all characters in a character set are printable. Character sets also include control characters. e.g. we have characters for a carriage-return, line-feed, form-feed, tab, bell etc. These are not having a print position, but give effect to position of next character or some such function.

In the initial days of computing each OS would support a particular character set and the most commonly supported character sets used to be either ASCII or EBCDIC(Extended Binary Coded Decimal Interchange Code). Later ASCII and its extensions were adopted by most of the OSs. The ASCII character set included only the commonly used latin characters and control characters. Later
ASCII and its extensions became more popular, and most new character sets were extensions of the ASCII character set. Each character set was catering to demands of a particular region/culture’s requirement of text representations. Most of these extensions of ASCII utilized the numeric values from 128 to 255 for the additional characters.

Some of the common examples of character sets which are extensions of ASCII are the ISO 8859 series, e.g. 8859-1 or Latin-1 caters to the Western European, 8859-2 or Latin-2 caters to Eastern European, 8859-3 or Latin-3 caters to Southern European, 8859-4 or Latin-4 caters to Northern European, 8859-5 or Cyrillic caters to Russian, Bulgarian, 8859-6 or Arabic, caters to the Arabic
characters, 8859-7 or Greek, caters to the Greek characters, 8859-8 or Hebrew, caters to Hebrew characters, 8859-9 or Latin-5 caters to Turkish characters, 8859-10 or Latin-6 caters to Northern European, 8859-11 or Thai, caters to the Thai characters, 8859-13 or Latin-7 caters to Baltic, 8859-14 or Latin-8 caters to Celtic, 8859-15 or Latin-9 caters to Western European and 8859-16 or Latin-10
caters to Eastern European. There other extensions of ASCII. We have ISCII (Indian Script Code for Information Interchange) which caters to the Indian scripts. The ISO 8859-12 was reserved for catering to the Devanagari script, but this was abandoned. The ISO 8859 series of character sets are summarized in Table below:

All the character sets mentioned above are extensioons of ASCII. i.e. They have the same characters in the range from 0 – 127, as ASCII. A quick summary of the ASCII characters is given in Table below:

The char data type used by many of the programming languages would simply rely on the OSs interpretation of the numeric value of the char data. i.e. if the OS used a different character set, the same numeric value would be interpreted differently. e.g. According to Latin-1 (8859-1) the value EB(Hex) is used for representing the character ë, whereas according to Greek (8859-7) the value EB(Hex) is used for representing the character

To solve this problem a universal character set was designed in the form of Unicode. The first version of Unicode was introduced in 1991. Unicode character set was designed to include all the characters available in all the languages/scripts of the world. This character set does get revised to include newer characters being added in various regions, as well as identification of some languages/scripts
which were not included in the earlier version. This character set has been designed to use numeric values from 0 — 0x10FFFFHex. This character set is also an extension of ASCII, so the initial values from 0 to 127 are same as ASCII. Most of the Indian scripts have been provided a block of 128 characters each, starting from 0x0900 onwards. The blocks for the Central and South East Asian Scripts in Unicode are summarized in Tables below:

What is the size of char in C? The size of char is usually 1 byte in C. The size of char in Java is 2 bytes. It is an unsigned integral, 16-bit value, used for representing UTF-16 code-units.

Why is the size of a char, 2 bytes in Java? In C, char represents a character from the platform’s local character set, which in most cases, is some extension of ASCII. The number of characters in most of these character sets is normally upto 256, so they require only 1 byte. In the case of Java, the char type is used to represent characters from the Unicode character set using the UTF-16 encoding, which requires 16 bits. The details of the UTF encodings will be given in another article.

Unicode

Let us understand Unicode. Unicode is a character set that has characters from all the languages of the world. There are various versions of the Unicode character set. At the time of writing this tutorial, the version of Unicode was 16. The Unicode standard maps characters from all the languages to a unique codepoint value. The codepoint values can be in the range of 0 —10FFFF (Hex). This codepoint range has been divided into 17 planes, each of 65536 values, i.e. 2^16 . The zeroeth plane, i.e. values from 0 —FFFF(Hex) is known as BMP (Basic Multilingual Plane), and other planes define the supplementary characters. To represent the complete range of characters using only 16-bit units, the Unicode standard defines an encoding called UTF-16. In this encoding, supplementary characters are represented as a pair of 16-bit code units, the first code unit from the high-surrogates range (D800 – DBFF(Hex)), and the second code unit from the low-surrogates range (DC00 – DFFF(Hex)). In Unicode standard, the range of codepoint values from D800 to DFFF (Hex) has not been assigned to any valid character and is reserved for surrogates. For characters in the range of 0000 —FFFF(Hex), the values of codepoints and UTF-16 code units are the same. The Java programming language represents text in sequences of 16-bit code units using the UTF-16 encoding. The char type in the Java programming language represents the 16-bit code unit.

Unicode Escapes in Java Source Code

The Java source code is a sequence of Unicode characters. The Java source code can contain characters from any language and not just characters from the ASCII character set. Most of the time the source code is encoded in some native character set, which is an extension of ASCII. Even in these cases the Java source code can include characters that are not part of the native character set. This is done by using the Unicode escape. In the source code we can specify any UTF-16 code unit by specifying the value as \u followed by four hexadecimal digits.

Tutorial to be continued for discusing literals and reference data types in Java.

About Identifiers in Java

Mon, 01 Sep 2025 12:01:47 +0000

Identifiers in Java

In the Java source code, we define/declare several entities. These entities are identifiable by some names. Identifiers are used to give names to the entities. These identifiers are used for naming the class(es), interface(s), enum(s), annotation(s), record(s). They are also used for giving names to various kinds of members of the class, interface, enum, annotation and record. The members which may be given names are the fields, methods. The identifiers are also used for giving names to variables. The identifiers are also used for giving names to the type parameters of a Generic class or method. The identifiers are also used for giving names to packages and modules. In case of package and module the names can use multiple identifiers separated by a '.' character. Identifiers are also used for giving names to labels within a method or a block. These labels are target of the break and the continue statements.

What are the rules for defining an identifier in Java? In Java, an identifier may contain any number of “Java letters” or “Java digits”, and it can start only with a “Java letter”. The sequence of “Java letters” in an identifier cannot match any of the keywords of the Java language or the boolean literals true, false or the literal null. A “Java letter” is not just the letters A — Z and a — z from the ASCII character set, but it also includes the letters from other languages available from the Unicode character set. The “Java letter” also includes the connecting punctuation characters like the ‘_’ character, currency symbols like the ‘$’, ‘₹’, ‘€’, ‘£’, ‘¥’, etc. sign of a numeric letter like the roman numeral ‘X’. The “Java letter or digit” also includes the digits used in the various languages available in the Unicode character set and not just the digits 0 — 9 from the ASCII character set. It also includes the combining marks and the non-spacing marks, which may be used for combining characters, The following declaration shows a valid declaration of a Java identifier:

char अ = 'अ';

Here अ has been used as an identifier; since it is a letter in Hindi, this declaration is valid. But then how do we use these characters in a Java source file, which is created using a text editor, where only the ASCII characters may be available? In a Java source file before the compiler identifies the lines and the tokens, it looks for Unicode escapes in the Java source file. The Java compiler works on Unicode characters. Our Java source file is normally encoded in ASCII or some extension of ASCII. While decoding from ASCII to Unicode, the compiler would first replace the Unicode escapes in the Java file with the actual Unicode character value. Using the Unicode escape we can write the above declaration in a Java source file encoded in ASCII as shown below:

char \u0905 = '\u0905'; // 0905 is the hex value for hindi letter A

Unicode escape is written as \u followed by four hexadecimal digits, where the hexadecimal digits are the codepoint values for that character in the Unicode character set. The following code segment would not compile:

char ch = '\u000A'; // 000A is value for line feed

since this will be seen by the Java compiler as:

char ch = '
';

Instead char ch ='\n'; should be used to have character literal for newline. Continuing with examples of valid and invalid identifiers for Java, the declaration

String नमस्ते = ”नमस्ते ”;

is valid in Java since it is only made up of Letters, but

String ९नमस्ते = ”नमस्ते ”;

is not valid since it starts with a digit(९ is Devanagari digit nine). But

String ₹९नमस्ते = ”नमस्ते ”;

would become valid since it now does not start with a digit, but a currency symbol.

Let us now define a class called नमस्तेदुनिया with a main method similar to the main method of the class HelloWorld, let use parameter name as आर्ग instead of args, and it should print “नमस्तेदुनिया” on the standard output instead of “Hello world”. This can be done as given below:

Listing of class नमस्तेदुनिया

class नमस्तेदुनिया {
    public static void main(String[] आर्ग) {
        System.out.println(”नमस्तेदुनिया” ) ;
    }
}

It is not a good idea to use non-ASCII text in the Java source code directly, since the interpretation of non-ASCII text would largely depend on the encoding used by the native OS. So, if we want to include non-ASCII characters in the Java source file, we may be better off using the Unicode escapes for all such characters. This may seem to be a tedious task. This can easily be taken care of by the native2ascii utility, which is part of JDK. This utility can convert any text file encoded using any of the standard encoding to ASCII by applying the Unicode escapes for all the non-ASCII characters, and can also be reversed back to native encoding using the same utility. e.g. if the code in the above Listing is saved in a file named HelloWorldHindi.java. And saved using the utf-8 encoding, then this Java source file can be converted to ASCII only, by using the command as given below:

native2ascii -encoding utf-8 HelloWorldHindi.java HelloWorldHindi.java

This file can be converted back to utf-8 encoded format with the help of command as given below:

native2ascii -encoding utf-8 -reverse HelloWorldHindi.java HelloWorldHindi.java

Note: Since Java 8, Java assumes that the default source files are encoded in utf-8. The native2ascii utility has been removed from JDK 9 onwards. If still want to use the utility then we can install JDK8 and use it from the bin folder of JDK8 installation.

Identifier ignorable characters

There are also some non-printable characters (most of these are control-characters), which are ignored in an identifier by the Java compiler. These characters are known as Java-Identifier-Ignorable. i.e. if any such character is used in an identifier it will be ignored, so it is possible to have two different sequence of Unicode characters in an identifier, which mean the same. e.g.

class TestIdentifierIgnorable {
    public static void main(String[] args) {
        String str = ”Hello world!” ;
        System.out.println(s\u0001tr) ; // \u0001 is Java Identifier Ignorable character
    }
}

In the above code listing, the variable name str in line 3 is same as s\u0001tr used in line 4. So, the above code is legal, compiles successfully and when run, would print “Hello world!” on the standard output.

The following are the numeric value in hex for the Java Identifier Ignorable characters.

0, 1, 2, 3, 4, 5, 6, 7, 8, e, f, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 1a, 1b, 7f, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 8a, 8b, 8c,42, 8d, 8e, 8f, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 9a, 9b, 9c, 9d, 9e, 9f, ad

600, 601, 602, 603, 604, 605, 61c, 6dd, 70f, 8e2, 180e, 200b, 200c, 200d, 200e, 200f, 202a, 202b, 202c, 202d, 202e, 2060, 2061, 2062, 2063, 2064, 2066, 2067, 2068, 2069, 206a, 206b, 206c, 206d, 206e, 206f, feff, fff9, fffa, fffb

110bd, 110cd, 13430, 13431, 13432, 13433, 13434, 13435, 13436, 13437, 13438, 1bca0, 1bca1, 1bca2, 1bca3, 1d173, 1d174, 1d175, 1d176, 1d177, 1d178, 1d179, 1d17a, e0001, e0020, e0021, e0022, e0023, e0024, e0025, e0026, e0027, e0028, e0029, e002a, e002b, e002c, e002d, e002e, e002f, e0030, e0031, e0032, e0033, e0034, e0035, e0036, e0037, e0038, e0039, e003a, e003b, e003c, e003d, e003e, e003f, e0040, e0041, e0042, e0043, e0044, e0045, e0046, e0047, e0048, e0049, e004a, e004b, e004c, e004d, e004e, e004f, e0050, e0051, e0052, e0053, e0054, e0055, e0056, e0057, e0058, e0059, e005a, e005b, e005c, e005d, e005e, e005f, e0060, e0061, e0062, e0063, e0064, e0065, e0066, e0067, e0068, e0069, e006a, e006b, e006c, e006d, e006e, e006f, e0070, e0071, e0072, e0073, e0074, e0075, e0076, e0077, e0078, e0079, e007a, e007b, e007c, e007d, e007e, e007f.

Note: In order to use any of the values above Hex FFFF(supplementary characters), it will have to be encoded using the UTF-16 encoding, which will require two char values. e.g. the value Hex 110BD will be encoded as two 16-bit values, which will be, Hex D804 and DCBD(These pair of values is known as surrogate pair, first value is high-surrogate and the second value is the low-surrogate). So in any identifier having a pair of character \uD804\uDCBD will be ignored by the compiler.

i.e. the identifier str is also equivalent to s\uD804\uDCBDtr.

Misconception Note:

Misconception - Letters used for forming an identifier are only from the ASCII character set i.e. A – Z and a – z.

Fact - Java uses Unicode character set, and so identifiers can use letters from any of the languages available in Unicode.

Misconception - Digits used for forming an identifier are only from the ASCII character set i.e. 0 – 9.

Fact - Java uses Unicode character set, and so identifiers can use digits from any of the languages available in Unicode.

Misconception - $ is the only currency symbol available for use in an identifier.

Fact In Java, identifier can use any currency symbol available in the Unicode character set.