Coder Legion - Recent posts tagged regex

Why grep "error|failed" finds nothing in a log full of failures

Thu, 22 Oct 2026 13:00:00 +0000

If grep returns empty on a file you know contains matches, suspect the pattern before the file. Default grep uses Basic Regular Expressions (BRE), where |, +, ?, and () are literal characters unless backslash-escaped. So grep "error|failed" app.log searches for the literal twelve-character string error|failed — pipe included — which no logger ever writes. Alternation needs Extended Regular Expressions via -E:

grep "error|failed" app.log     # BRE: literal string, matches nothing
grep -E "error|failed" app.log  # ERE: matches either word

The dangerous part is the exit behavior. An impossible pattern is not a syntax error, so grep exits 1 — the same code as "no matches in a clean file" — and prints nothing. A wrong question and a healthy log look identical, which is how this bug hides inside monitoring checks and cron pipelines for days.

Two related traps: -P (Perl regex, \d, lookaheads) is GNU-only and fails on macOS/BSD grep, and bare grep aimed at a directory refuses with "Is a directory" unless -r is set.

When I'm unsure what a pattern will match, I assemble the command in this grep command builder with a live match tester — paste a sample line you know should match, and a zero count tells you the pattern is the bug before the empty result gets believed.

Regular Expression - A Comprehensive Guide

Mon, 02 Jun 2025 23:03:31 +0000

(Originally posted on Methodox Wiki, visit the page for a live tester!)

Contents

Introduction
Live Tester
Practical Syntax
Comprehensive Reference
Conclusion

(Due to CoderLegion content length limit, section 4 is ommitted)

1. Introduction

Text - whether it’s log files, configuration data, user input, or documents - is everywhere in programming. From simple scripts to full-blown applications, developers frequently need to search, validate, extract, or transform pieces of text. While many IDEs and editors (like Notepad++, VS Code, or Sublime Text) offer basic find-and-replace functionality, they quickly reach their limits when you need to handle patterns, conditional logic, or repeated structures. That’s where regular expressions (regex) come in.

At its core, a regular expression is a compact, declarative way to describe a set of strings. Instead of manually scanning through text to find, say, all warning messages in a log, you can write a single pattern that matches timestamps, log levels, filenames, email addresses, URLs - you name it. Once you understand the fundamentals, regex can become one of the most powerful “weapons” in your developer toolbox.

Consider a typical log snippet:

2025-05-31 14:23:48 [Info] Starting application...
2025-05-31 14:23:50 [Info] Loading application data...
2025-05-31 14:23:55 [Warning] Missing font file: New Times Roman.
2025-05-31 14:24:22 [Info] The application has started, with 1 warning.
2025-05-31 14:25:01 [Error] Unhandled exception: NullReferenceException.

If you want to answer questions like:

“What warnings were emitted, and when?”
“Which lines contain error messages?”
“How many times did the application start?”

You could write a script in your favorite language that splits on spaces, checks tokens, and does string comparisons. But that quickly becomes tedious: parsing variability in timestamps, handling optional fields, or ignoring whitespace quirks can make your code fragile.

With a well-crafted regex, however, you can match on the entire log line, capture the timestamp, the log level, and the rest of the message - all in one pattern. More importantly, regex can handle any text-based format for which there’s a consistent structure: CSV files, HTML/XML tags, custom configuration formats, and so on.

By the end of this article, you will:

Understand the basic motivation behind regular expressions.
See practical syntax elements and examples for day-to-day tasks.
Have a handy reference that summarizes the most commonly used regex constructs.

Whether you’ve done a bit of “find and replace” in an editor, or you’ve written simple string processing code, this guide will bridge the gap and show you how to level up with regular expressions.

2. Live Tester

See original post.

3. Practical Syntax

In this section, we’ll cover the core building blocks of regex patterns. Each subsection introduces a concept, followed by real-world examples - some based on log processing, others on everyday tasks like validating email addresses or extracting numbers from text.

3.1 Literal Characters and Simple Matches

At the simplest level, a regex can be just literal characters. For example:

Pattern Error will match any occurrence of the substring “Error” in a text.
Pattern 2025-05-31 will match that exact date string.

However, real-world text rarely lines up perfectly. Log messages, for instance, include varying timestamps and different log levels. To match them in a generic way, you need metacharacters.

Key Point: Most characters in a regex match themselves literally, except for special metacharacters (like ., *, ?, \, etc.). To match a literal period, bracket, or other metacharacter, you must “escape” it with a backslash (\. to match a literal dot, \[ to match a literal left bracket, and so on).

3.2 Character Classes (`[]`) and Ranges

A character class is a bracketed set of characters that match exactly one character from the set. For example:

[AEIOU] matches any single uppercase vowel.
[0-9] matches any digit from 0 through 9.
[A-Za-z0-9_] matches a typical word character (letter, digit, or underscore).

Example 1: Matching a Four-Digit Year

Suppose you want to match any year between 1900 and 2099:

^(19|20)[0-9]{2}$

^ and $ are anchors (we’ll cover them in a moment).
(19|20) matches either “19” or “20”.
[0-9]{2} matches exactly two digits.
Altogether, this matches “1900” through “2099”.

Example 2: Extracting Log Levels

Given a log line like:

2025-05-31 14:23:55 [Warning] Missing font file: New Times Roman.

You can match the log level (Info, Warning, Error, etc.) with:

\[(Info|Warning|Error|Debug)\]

\[` and `\] match literal square brackets.
(Info|Warning|Error|Debug) matches one of those four words.

If you expect additional levels (e.g., Trace or Fatal), you can add them inside the parentheses:

\[(?:Info|Warning|Error|Debug|Trace|Fatal)\]

Here we’ve used (?: … ) to create a non-capturing group (we’ll talk about groups more in section 5).

3.3 Quantifiers: Repetition Control

Quantifiers specify how many times the preceding element should match. The most common quantifiers are:

* – Match zero or more times (greedy).
+ – Match one or more times (greedy).
? – Match zero or one time (makes the preceding token optional).
{n} – Match exactly n times.
{n,m} – Match between n and m times (inclusive).
{n,} – Match n or more times.

Example 3: Matching Timestamps

A timestamp like 2025-05-31 14:23:48 can be matched with:

\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}

\d is shorthand for [0-9].
\d{4} matches exactly four digits (the year).
- matches a literal hyphen.
\d{2} matches two digits (month and day).
\s+ matches one or more whitespace characters (space, tab).
\d{2}:\d{2}:\d{2} matches HH:MM:SS.

If you’re sure there’s exactly one space between date and time, you could use \s instead of \s+.

Example 4: Optional File Extension

Suppose you have filenames like report.txt or report. To match both, you could use:

^report(?:\.txt)?$

^ and $ ensure we match the entire string.
(?:\.txt)? means “optionally match ‘.txt’”.
- \. matches a literal dot.
- txt matches “txt”.
- The ? after the group makes it optional.

3.4 Anchors: Positioning Matches

Anchors specify positions in the text rather than actual characters:

^ – Start of the line/string.
$ – End of the line/string.
\b – Word boundary (between \w and \W).
\B – Non-word boundary.

Example 5: Lines Starting with “Error”

If you want to catch any line that begins with the word “Error” (as a standalone word), you can write:

^Error\b.*$

^Error ensures the line starts with “Error”.
\b asserts a word boundary so that “Errors” or “ErrorCode” are not matched.
.* matches the rest of the line (any character, zero or more times).
$ anchors the end of line.

3.5 Groups and Capturing

Parentheses ( and ) serve two main purposes:

Grouping: Treat multiple tokens as a single unit (often used with quantifiers or alternation).
Capturing: Store the text matched by that group for later retrieval (in code or in a replacement string).

Example 6: Capturing Timestamps and Levels

Given:

2025-05-31 14:23:55 [Warning] Missing font...

You might write:

^(\d{4}-\d{2}-\d{2})\s+(\d{2}:\d{2}:\d{2})\s+\[(Info|Warning|Error|Debug)\]\s+(.*)$

(\d{4}-\d{2}-\d{2}) captures the date (Group 1).
(\d{2}:\d{2}:\d{2}) captures the time (Group 2).
(Info|Warning|Error|Debug) captures the log level (Group 3).
(.*) captures the rest of the message (Group 4).
In most languages, after matching, you can extract Group 1, Group 2, etc.:
- e.g., in C#: match.Groups[1].Value gives the date.

If you want to group without capturing (for alternation or quantifier purposes), use (?: … ). For instance:

(?:Info|Warning|Error|Debug)

does not create a numbered backreference.

3.6 Common Metacharacters and Escape Sequences

Some of the most frequently used shorthand and metacharacters include:

. – Matches any character except newline (in most flavors; some allow a “dotall” mode to match newline).
\d – Digit (equivalent to [0-9]).
\D – Non-digit (equivalent to [^0-9]).
\w – Word character (letter, digit, underscore; [A-Za-z0-9_]).
\W – Non-word character (anything not in \w).
\s – Whitespace character (space, tab, newline, etc.).
\S – Non-whitespace character.
\t, \n, \r – Tab, newline, carriage return.

Tip: When writing regexes in code, you often need to escape backslashes. For instance, in C# you can use verbatim string literals (@"\d{4}-\d{2}-\d{2}") to avoid double escapes.

3.7 Practical Example: Extracting Email Addresses

Say you have a document or a configuration file and need to pull out all email addresses. A simple - but not fully RFC‑compliant - pattern is:

\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b

\b – Ensure we start at a word boundary.
[A-Za-z0-9._%+-]+ – One or more letters, digits, dots, underscores, percent signs, plus, or hyphens (the local part before the @).
@ – Literal “@” symbol.
[A-Za-z0-9.-]+ – One or more letters, digits, dots, or hyphens (domain).
\. – Literal dot.
[A-Za-z]{2,} – At least two letters for the TLD (e.g., “com”, “org”, “io”).
\b – End at a word boundary to prevent partial matches (so we don’t match trailing punctuation).

Testing It

Given:

Please contact *Emails are not allowed* or *Emails are not allowed* for more info. Alternatively, reach out to admin@localhost if testing locally.

*Emails are not allowed* matches.
*Emails are not allowed* matches.
admin@localhost does not match because localhost has no “. + two-or-more-letters” TLD.

3.8 Practical Example: Validating North American Phone Numbers

North American Numbering Plan (NANP) phone numbers often look like:

123-456-7890
(123) 456-7890
123.456.7890
+1 (123) 456-7890

A practical regex might be:

^(\+1\s?)?           # Optional country code +1 and a space
(?:\(\d{3}\)|\d{3})  # Either (123) or 123
[ .-]?               # Separator: space, dot, or hyphen (optional)
\d{3}                # Next three digits
[ .-]?               # Separator again (optional)
\d{4}$               # Last four digits

When compacted (and removing whitespace/comments):

^(\+1\s?)?(?:\(\d{3}\)|\d{3})[ .-]?\d{3}[ .-]?\d{4}$

^ and $ anchor the pattern to the entire string.
(\+1\s?)? – Optionally match “+1” followed by an optional space.
(?:$\d{3}$|\d{3}) – Either three digits inside parentheses or three digits without.
[ .-]? – Optionally match a space, a dot, or a hyphen.
\d{3} – Three digits.
[ .-]? – Separator again.
\d{4} – Four digits.

This pattern will match most common North American formats. By tweaking the groups and quantifiers, you can adapt it for other regions or stricter/looser rules.

3.9 Practical Example: Parsing Log Files

Returning to our log example, suppose you want to extract only the warning messages and their timestamps. You might apply a regex like:

^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2})\s+\[Warning\]\s+(.*)$

Group 1 captures the timestamp (2025-05-31 14:23:55).
Group 2 captures the message itself (Missing font file: New Times Roman.).

In most programming languages:

C# (using System.Text.RegularExpressions):

var pattern = @"^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2})\s+\[Warning\]\s+(.*)$";
var matches = Regex.Matches(logContents, pattern, RegexOptions.Multiline);
foreach (Match m in matches) {
    string timestamp = m.Groups[1].Value;
    string message   = m.Groups[2].Value;
    Console.WriteLine($"[{timestamp}] Warning: {message}");
}

Python (using the re module):

import re

pattern = r'^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2})\s+\[Warning\]\s+(.*)$'
for line in log_contents.splitlines():
    m = re.match(pattern, line)
    if m:
        timestamp = m.group(1)
        message   = m.group(2)
        print(f"[{timestamp}] Warning: {message}")

Even if you’ve only ever clicked “Replace” in an editor, these examples should give you a clear sense of how powerful a single regex can be. You define “match everything up until [Warning], then capture whatever comes next”, and you get exactly the data you want.

3.10 Testing and Debugging Regex

When you’re new to regex, it’s easy to overlook small mistakes - missing an escape, misplacing a quantifier, forgetting an anchor. Here are a few tips:

Use an Online Tester
Websites like regex101.com (PCRE, JavaScript, Python), RegexStorm (for .NET), or regexr.com let you type your pattern, paste test text, and see matches/highlights in real time.
Start Simple, Then Grow
If you need a complex pattern, start with a small piece. For instance, test just \d{4}-\d{2}-\d{2} on the date first, then add time, then add log level, etc.

Comment Your Patterns (When Supported)
Some languages/flavors support the “extended” or “free-spacing” mode, where you can add whitespace and # comments inside your regex:

(?x)            # Enable free-spacing mode
^               # Start of line
(\d{4}-\d{2}-\d{2})   # Group 1: Date
\s+             # One or more spaces
(\d{2}:\d{2}:\d{2})   # Group 2: Time
\s+             # One or more spaces
\[Warning\]     # Literal [Warning]
\s+             # One or more spaces
(.*)            # Group 3: Message text
$

This makes maintenance and future edits easier, especially for complex regexes.

Escape Early
If you need to match special characters (., *, ?, +, [, ], (, ), |, ^, $, \), always escape them with a backslash unless you explicitly want their special behavior.

4. Comprehensive Reference

Omitted due to content length limit. Visit Methodox Wiki for more.

5. Conclusion

By now you should have a solid grasp of:

Why regular expressions are invaluable for text processing.
How to write basic patterns: literals, character classes, quantifiers, anchors.
How to use groups, lookarounds, and backreferences for more advanced matching.
Which tokens and constructs constitute the “core” of most practical regexes.
Where to look when you need to refresh your memory (the reference section above, or a site like regexstorm.net/reference).

With a few dozen minutes of practice - writing regexes against sample text - you’ll rapidly gain confidence. Remember:

Test early and often in an online tester or your IDE’s “Find in Files” - seeing the matches in real time is far more enlightening than manually eyeballing code.
Readability matters. If the pattern becomes unreadable, use the free-spacing mode ((?x)) and add comments.
Keep it simple for common tasks. You rarely need the most arcane syntax; matching log levels, dates, email addresses, or basic CSV fields can usually be done with straightforward character classes and quantifiers.

Once you’ve mastered the basics, you can explore more advanced topics like:

Atomic grouping and possessive quantifiers (for performance-critical patterns).
Recursive patterns (in PCRE or some advanced flavors) to match nested constructs.
Conditional subpatterns (in PCRE or .NET) where you test if a group has matched before deciding the next part of the pattern.

But those are topics for another day. For now, lean on the examples and reference sheet above, and try applying regex to your next text‑processing challenge. In no time, you’ll find yourself thinking in patterns - and you won’t look back.

Congratulations on starting your pattern crafting journey! Keep this guide handy, bookmark your favorite regex tester, and remember that a well‑written regular expression often turns hours of manual parsing into a single line of elegant code.

References

RegStorm for C#

Regular Expressions in Python

Sat, 06 Apr 2024 00:24:25 +0000

Mastering Regular Expressions in Python

I. Introduction

We all have been regularly searching for keywords in our favourite articles, and also using the "find and replace" feature in our text editors to correct mistakes. All this is done using Regular Expressions, which is the topic of discussion in this article. Regular Expressions or RegEx is a sequence of characters which specify a certain search pattern of strings. Regular Expressions are an integral part of all text processing tasks in text editors and also for text validation of user inputs. In this article, we shall discuss all about regular expressions and practice using the Python ‘re’ module

II. Understanding Regular Expressions

Regular Expressions or RegEx can be defined as a sequence of characters which form a pattern that can be searched in strings. The characters in a search pattern could be numbers, alphabets, non-alphanumeric characters. We may even want to search for characters at certain positions like the starting or end of a string or at word boundaries. There may be conditions about the number of times we want those characters in the pattern to be repeated. All such conditions help us define our regular expressions.

Basic syntax and components of regex patterns:

import re
print(re.search(r"^[cd]o{2}.ing\d","cooding2"))

RegEx comprise the following components:
1) Literals: The characters which match themselves in the string are known as Literals. Suppose we want to match “coding”, then our pattern could simply be “coding”. Here all the characters are literals, with no special meaning attached to any one of them.

2) Metacharacters: Some characters have special meanings in a regular expression. They are known as Metacharacters.
Examples:

. "period" is a special character which represent all characters except new line characters. Pattern .at matches “cat”, “bat”, “mat” etc.
* "asterisk" represents zero or more occurrences of the character preceding it. Pattern ra*t matches “rt”, “rat”, “raat”, “raaat”
+ "plus" represents one or more occurrences of the preceding element. Pattern ra+t matches “rat”, “raat”, “raaat” but not “rt”
? "question mark" represents zero or one occurrence of the preceding element. Pattern ra?t matches “rat” and “rt” but not “raat”
^ "caret" represents the start of the string. So, the pattern “^a” will match the a in “air” but not in “fair”
$ "dollar sign" represents the end of the string. So, the pattern d$ will match the d in “sad” but not in “fade”.
\ "backslash" Helps us escape a metacharacter. Suppose we want to search for a price of item which is 200$ . But in regular expressions $ is metacharacter. The pattern “200$” will search for a pattern of 200 at the end of string. So, if we want the regular expression to treat $ as a literal, we will escape its metacharacter meaning using a “\” symbol. So, we will make our regular expression as “200\$”

Special sequence: The symbol \ in combination with some characters indicate a special sequence
•\d matches digits 0-9,
•\w matches word characters 0-9, a-z, A-Z, and underscore “_”
•\s matches any whitespace character (space, tab, newline).
•\b matches a word boundary (position between a word character and a non-word character).
•\D matches any character that is not a digit.
•\W matches any character that is not a word character.
•\S matches any character that is not a whitespace character.

3) Character Classes: Suppose we want the character in our search pattern to be any one of a set of characters. Then we may use a character class. For example, we may want our word to be “bat” or “mat” then our regular expression will look like: [bm]at. Here [] will represent the character class, which includes the characters b and m and any one of these will match our pattern.

4) Quantifiers: Suppose we want to specify that our pattern must contain exactly "n" consecutive occurrences of a sub pattern or character. We may use quantifiers in such cases. The quantity is specified inside{} curly braces. We want to match 3 occurrences of “a”. Our Pattern would be a{3}

5) Anchors: These specify the positioning of the characters in a pattern. We have already discussed these anchors like

^ start of string
$ end of string
\b word boundary

III. Getting Started with re Module

Python offers a module by the name of ‘re’, which provides regular expression matching operations. Both Unicode strings (str) as well as 8-bit strings (bytes) can be searched using this module.

Importing the `re` module:

Python ‘re’ module comes as part of the ‘The Python Standard Library’. It is included as a module under the “Text processing services” of the Python Standard Library. So, we do not need to install it separately using pip command like other packages. Rather we can just import the module and use it.

import re

Overview of key functions and methods in the `re` module:

The entire documentation of the re module can be viewed at the official website. Here we will discuss the commonly used functions and methods of the re module.
re.search :It search a string for the presence of a pattern. Only the first match will be returned.
re.findall :It will find all occurrences of a pattern in a string and return them as a list.
re.finditer:It will return an iterator yielding a Match object for each match.
re.compile :It will compile a pattern into a Pattern object. Using a Pattern object is much more efficient if the same regular expression has to be used again and again.
re.match : It will match a regular expression pattern to the beginning of a string.
Let see a demonstration of these functions.

import re

my_text = "do you code daily, code daily to become an expert. I do code a lot"

print(re.search(r'code', my_text)) # only the first occurence is matched

print(re.findall(r'code', my_text)) # all 3 occurences are matched

m = re.finditer(r'code', my_text) # an iterator object is created 
print(m)
for individual_match in m: # loop over the iterator object to see all matches
    print(individual_match)
    
my_pattern = re.compile(r'do')
my_pattern1 =re.compile(r'you')
print(my_pattern.match(my_text)) # the pattern at the beginning of the string matched
print(my_pattern1.match(my_text))

Regular Expression Objects:
Pattern object : It is a compiled regular expression object returned by re.compile(). It offers methods like Pattern.match and Pattern.search.
Match object: It is returned by successful matches and searches.
It offers methods like Match.group.

Note: Always use the Python's raw string notation r"your_pattern" for Regular Expressions to avoid unexpected behaviour. Raw strings allow you to use backslashes \ without needing to escape them. Unlike regular strings, where backslashes have special meanings like \n representing a new line, within a raw string, \ retains its literal value.

IV. Basic Regex Patterns and Operations

Let us now practically explore some of the concepts we have learnt till now.

import re

my_text = """when I start writing, I understand that when
cat and bat are two different words / 1 and 9 
are different digits in maths!"""

# Matching literal characters
print(re.findall(r'cat',my_text))

# Matching character classes without range
print(re.findall(r'[cb]at',my_text))

# Matching character classes with range
print(re.findall(r'[b-m]at',my_text))

# character class inversions using Caret ^ inside square brackets
print(re.findall(r'[^\w\s]',my_text))

#Quantifiers for matching repetitions
print(re.findall(r'f{2}',my_text))

#Anchors for boundary matching
print(re.findall(r'\b[b-m]at\b',my_text)) # word boundary
print(re.findall(r'^when',my_text)) # string boundary

A.Matching literal characters:

Here we start by trying to find the simple literal keyword “cat” in a string called “my_text”.

B. Using character classes and ranges:

• Next, we try to match either cat or bat by using character class where we enclose c and b inside the square bracket [cb]. That means if either c or b is present it will be a match.
• Taking this a step further we specify a range of characters, [b-m] any character between this range including b and m will match.
• We could even say that any character except the characters present in the character class by using inversion with a ^ Caret sign inside a square bracket

C. Quantifiers for matching repetitions:

We could use quantifiers which specify the exact number of times a character must repeat inside the curly bracket {}. Here we search for “f” repeated twice

D. Anchors for boundary matching:

Here we search for the pattern with word boundary specified with \b . Note that now “mat” in “maths” and “hat” in “that” do not match as “mat” will not have a word boundary after it and “hat” will not have a word boundary before it.
Similarly, ^ outside of a character class represents a string beginning, so only one “when” at the beginning of the string will be matched.

V. Advanced Regex Patterns

A. Grouping and capturing in regular expressions:

When we find patterns using regular expressions, we can create groups of a set of characters by enclosing them in parenthesis (). These groups can then be accessed and processed separately. We may even specify our groups as non-capturing using the notation(?:pattern). Also, we can name our groups as well using the notation (?P<name>pattern) and then access them using those names. Let’s look at an example:

Suppose we have a key in our data which looks like “CODE-NAME-DEPT-SALARY-EXPERIENCE”. We can create a regular expression to match this pattern. But if we want to further process the salary and the experience to find the average salary and average experience level of people, then we will create groups for these. We can specify the Name and the age as non-capturing groups here. Let’s look at the example.

import re
salary_list = []
experience_list = []
# CODE-NAME-DEPT-SALARY-EXPERIENCE is the key
my_data = "CODE-Sam-SALES-2300-2 CODE-Tina-TESTING-2600-4 666 CODE-Divya-MARKETING-10000-12 hello"
# pattern1 has 2 non-capturing groups and 2 named groups
pattern1 = r'CODE-(?:\w+)-(?:\w+)-(?P<salary>\d+)-(?P<experience>\d+)'
pattern2 = r'CODE-(\w+)-(\w+)-(\d+)-(\d+)'
m1 = re.finditer(pattern1, my_data)
m2 = re.finditer(pattern2, my_data)
for m in m1:
    print(f"group 1 is for salary: {m.group(1)}")
    print(f"group 2 is for experience: {m.group(2)}")
    salary_list.append(int(m.group('salary')))
    experience_list.append(int(m.group('experience')))
print(f"Average salary is : {sum(salary_list)/len(salary_list)}")
print(f"Average experience level is : {sum(experience_list)/len(experience_list)}")

B. Alternation and conditional matching:

Regular expressions can also be used to match alternate patterns. Then depending on which pattern has been captured we can further conditionally match other patterns. The | operator is used for alternation and the (?(id/name)yes|no) syntax could be used for conditional matching. Let us suppose we want to capture a word say “one” but we have two different languages. If it is English we capture “one”, if it is French we capture “un”.

import re
my_text = "ENGLISH:one"
my_text1 = "FRENCH:un"
my_text2 = "FRENCH:one"
pattern = r'((ENGLISH)|(FRENCH)):(?(2)(one)|(un))'
m = re.search(pattern, my_text)
m1 = re.search(pattern, my_text1)
m2 = re.search(pattern, my_text2)
print(m)
print(m1)
print(m2)

C. Lookahead and lookbehind assertions:

We can check if our regular expression pattern we are searching for has or does not have an expression before it or after it. Let the pattern to be searched be called X, and the expression to be checked before or after it be called Y. The expression Y would not be included as part of the result. The syntax for this would be:
Positive Lookahead : X(?=Y)
Negative Lookahead: X(?!Y)
Positive Lookbehind : (?<=Y)X
Negative Lookbehind: (?<!Y)X
Let us look at an example:

import re
my_text = "onehello1 zerohello2 hello3one hello4zero"
pos_lookahead = r'(hello\d)(?=one)' # r'hello\d' followed by one
neg_lookahead = r'(hello\d)(?!one)' # r'hello\d' not followed by one
pos_lookbehind = r'(?<=one)(hello\d)' # r'hello\d' preceeded by one
neg_lookbehind = r'(?<!one)(hello\d)' # r'hello\d' not preceeded by one 
print(re.findall(pos_lookahead, my_text))
print(re.findall(neg_lookahead, my_text))
print(re.findall(pos_lookbehind, my_text))
print(re.findall(neg_lookbehind, my_text))

D.Backreferences:

Backreferences in a pattern allow us to specify that the contents found in an earlier capturing group must also be found at the current location in the string. Let us suppose we want to find the words which have the same starting and ending letter. We could then use backreference to capture such words.
Previously captured groups can be referred to by using backslash followed by the group number (e.g., \1, \2, \3, etc.)

import re
my_text = "In this harsh world , when the going gets tough the tough gets going"
pattern = r'\b(\w)\w*\1\b' # will match words which begin and end in the same letter
all_matches =(re.finditer(pattern, my_text))
for match in all_matches:
    print(match)

VI. Practical Examples and Use Cases

A. Validating input data

Suppose, we wish to validate the user input form. It may contain pin code. We can create a regular expression and use it to validate the particular user input. Suppose we wish to validate the Indian PIN code. Let’s keep the validation rules simple:
PIN code is exactly six-digit long code.
The first digit is 1-9, all other digits can be 0-9. (There are exceptions though, as some of the PIN codes might not exist.)
Regular expression pattern will be “[1-9]\d{5}”
Now we can validate the user input PIN code.

import re
entered_pincode = input("please input PINCODE\n")
pattern = r'^[1-9]\d{5}$'
if re.match(pattern, entered_pincode):
    print("valid PINCODE")
else: 
    print("Invalid PINCODE")

B. Extracting information from text (e.g., parsing log files)

Regular expressions can be used to extract information from text, like from log files.
Suppose my Log file contains information about users who have logged into a system. Let us try to extract the name of the person who logged into the system as well as the date.

import re

my_login_info = "INFO:UserA logged into system on date: 04-11-2024"
pattern = r'INFO:(\b.+\b) logged into system on date: (\b\d{2}-\d{2}-\d{4}\b)'
match = re.search(pattern, my_login_info)
print(match)
if match:
    print(f"The following user logged into the system: {match.group(1)}")
    print(f"Date of Login: {match.group(2)}")

The extracted information can then be used for further interpretation and analysis.

C: Search and replace operations in text processing tasks

Suppose we used an incorrect word in a paragraph. We could find and replace every occurrence of that word and substitute it with the correct word.
Let’s say we used weak instead of week and our word editor could not spot this mistake, since it cannot understand the context. Let’s correct this mistake with our regular expression.

import re

my_paragraph = "Decorators in weak 32.What do you suppose I should be studying in weak 33? Any tweak?"
misspelled_word = r'\bweak\b'
my_paragraph = re.sub(misspelled_word, "week", my_paragraph)
print(my_paragraph)

Note that here tweak was not changed as we had used word boundaries in our regular expression pattern

D. Filtering and cleaning textual data using regular expressions

We need to filter and clean our data on a regular basis, so that some further processing can be performed on it.
For example, suppose we want to find the most frequently used words in a book. The first step would be to clean the text of the book, removing all the special characters, replacing the new line characters “\n”, tabs “\t” etc with only a single space. This can be done using the regular expressions.
Let’s see an example for the same.

import re

my_book = "This happens    to be a very **small** book written in <English>, by a renowed author!!!"
my_book = re.sub(r"[^\w\s]", "", my_book)
print(my_book)
my_book = re.sub(r"\s+", " ", my_book)
print(my_book)
my_word_list = my_book.split(" ")
print(my_word_list)

So, here we have created a list of words in the book after cleaning the text of the book.

VII. Tips and Best Practices

Tip: Compile your regex patterns using re.compile if the same pattern is to be used multiple times
Use character classes whenever possible.
Use non-capturing groups if you do not require to further process them.
Do not forget to escape special characters if you want to match them as Literals
Use flags like re.IGNORECASE if you want your search to be case-insensitive

VIII. Conclusion

The knowledge of Regular Expressions is indispensable for text processing, data manipulation, and pattern matching tasks in real world applications. The Python re module offers many functionalities related to regular expressions usage. I hope this tutorial provides you with enough information to make you curious enough to explore the power of regular expressions in depth. I would encourage you to read the official documentation of the re module and explore more functions, methods and flags related to regular expressions. Only practice can help you master Regular expressions!