Python Regular Expressions (Regex)

Master regex basics to extract, match, and clean patterns from real-world unstructured text data.

6 minutesVideo LessonPDF notes

Watch Python Regular Expressions (Regex) on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ready to continue?

Mark this lesson as complete when you're ready to proceed.

Key moments

Introduction to Regex — Regular expressions are a powerful, precise tool for finding patterns in text, useful for editing code or validating input.
Testing Patterns — You can test regex patterns in a text editor's search function using symbols like \d for digits and \D for non-digits.
Importing RE Module — The regular expressions package in Python is named RE, which must be imported before use.
Email Validation Pattern — A complex pattern is constructed using character sets, the plus sign quantifier, and escaped periods to validate email structure.
Using re.search — The `re.search` function checks if the defined email pattern is found within the user's input string.
Replacement Goal — The second example aims to remove hyphens from phone numbers while leaving hyphens in words untouched.
Grouping and Raw Strings — A phone number pattern is defined using capturing groups and the raw string prefix `r` to handle backslashes correctly.
Applying re.sub — The `re.sub` function uses backreferences to replace the hyphenated phone number with a continuous string of digits.

PDF notes

Bottom line: You will be able to use Python's `re` module to define precise text patterns for data analysis tasks. You can validate user input like emails and perform large-scale string replacements in databases. This fundamental skill is universal across programming languages.

This lesson on Python Regular Expressions (Regex) is hands-on and example-driven. You will be able to use Python's re module to define precise text patterns for data analysis tasks. You can validate user input like emails and perform large-scale string replacements in databases. This fundamental skill is universal across programming languages.

What You'll Be Able To Do

Import the Python re module for pattern matching operations.
Construct basic regular expressions using character classes like \d and [].
Validate user input strings, such as emails, using re.search().
Define and apply complex replacement patterns using grouping and re.sub().
Test regex patterns quickly using text editor search functions (e.g., Ctrl+F).

Detailed Concept Walkthrough

1. Regular Expression Basics

Regex is a powerful syntax for finding and manipulating specific text patterns, acting as a precise version of 'Ctrl+F'. Symbols are universal across programming languages, making it a transferable skill.

Mechanism: The re module in Python provides functions like search and sub to apply defined patterns to target strings.
Syntax Rule: Character classes like \d (digits 0-9) and \D (non-digits) simplify pattern definition; uppercase often represents the inverse of the lowercase symbol.
Best Practice: Use raw strings (r"pattern") to prevent Python from interpreting backslashes as escape sequences before the regex engine sees them.

import re
# \d matches digits (0-9)
pattern_d = r"\d+" 
text = "Item 102"
match_d = re.search(pattern_d, text)
# print(match_d.group(0)) # Output: 102

Key Takeaway: Regex symbols define character sets and repetition rules to match complex structures in text.

2. Validating Input with re.search

Email validation ensures user input adheres to a required format (e.g., local-part@domain.tld) before processing. The re.search() function checks if the pattern exists anywhere in the string.

Mechanism: Square brackets [] define a custom character set (e.g., [a-zA-Z0-9]). The + quantifier requires one or more occurrences of the preceding element.
Execution Flow: re.search(pattern, string) returns a match object if the pattern is found, which evaluates to True; otherwise, it returns None (evaluates to False).
Syntax Rule: To match a literal period (.), it must be escaped using a backslash (\.) because the period is normally a special regex character matching any character.

import re
EMAIL_PATTERN = r"[a-zA-Z0-9]+@[a-zA-Z]+\.(com|edu|net)"
user_input = "test@example.com"
if re.search(EMAIL_PATTERN, user_input):
    # Valid email logic here
    pass

Key Takeaway: Use re.search and character sets ([]) combined with quantifiers (+) to enforce structural requirements on input data.

3. String Replacement with re.sub

re.sub() allows you to find all occurrences of a complex pattern and replace them with a specified replacement string, crucial for data normalization.

Mechanism: Parentheses () create capturing groups. These groups capture the matched text segments within the pattern.
Functionality: re.sub(pattern, repl, string) uses the replacement string (repl) which can reference captured groups using backreferences (\1, \2, etc.).
Best Practice: Use groups to capture the desired segments (like number blocks) and reconstruct the string in the replacement step without unwanted separators (like hyphens).

import re
# Captures three groups of digits
phone_pattern = r"(\d{3})-(\d{3})-(\d{4})"
# Concatenates groups 1, 2, and 3 without hyphens
replacement = r"\1\2\3" 

data = "555-123-4567"
cleaned_data = re.sub(phone_pattern, replacement, data)
# print(cleaned_data) # Output: 5551234567

Key Takeaway: Capturing groups () and backreferences (\1) enable precise, conditional replacement of structured text segments.

Topics Covered in Python Regular Expressions (Regex)

Introduction to Regex (0:00 - 0:45) — Regular expressions are a powerful, precise tool for finding patterns in text, useful for editing code or validating input.
Testing Patterns (0:45 - 1:30) — You can test regex patterns in a text editor's search function using symbols like \d for digits and \D for non-digits.
Importing RE Module (1:30 - 2:00) — The regular expressions package in Python is named RE, which must be imported before use.
Email Validation Pattern (2:00 - 4:45) — A complex pattern is constructed using character sets, the plus sign quantifier, and escaped periods to validate email structure.
Using re.search (4:45 - 5:40) — The re.search function checks if the defined email pattern is found within the user's input string.
Replacement Goal (6:45 - 7:45) — The second example aims to remove hyphens from phone numbers while leaving hyphens in words untouched.
Grouping and Raw Strings (7:45 - 8:45) — A phone number pattern is defined using capturing groups and the raw string prefix r to handle backslashes correctly.
Applying re.sub (8:45 - 9:30) — The re.sub function uses backreferences to replace the hyphenated phone number with a continuous string of digits.

Python Cheat Sheet

import re — Access Python's built-in regex functions
```
import re
```
re.search(pattern, string) — Checks if pattern exists in string
```
re.search(r"\d", "a1")
```
re.sub(pattern, repl, string) — Replaces pattern matches with replacement
```
re.sub(r"-", "", "1-2")
```
r"pattern" — Defines a raw string for regex patterns
```
pattern = r"\d+"
```
\d — Matches any single digit (0 through 9)
```
re.search(r"\d", "10")
```
\D — Matches any single character that is not a digit
```
re.search(r"\D", "A1")
```
[a-zA-Z0-9] — Matches any character within the specified set
```
re.search(r"[A-Z]", "Hi")
```
+ — Matches one or more occurrences of preceding element
```
re.search(r"a+", "aaab")
```

Comparison Table

Symbol	Matches	Purpose
\d	Any digit (0-9)	Find numerical characters
\D	Any non-digit character	Find non-numerical characters
()	Capturing Group	Extract or reference segments

Common Pitfalls

Mistake: Forgetting to import the re package before using functions. Avoid: Always start your script with import re.
Mistake: Not escaping special characters like the period (.). Avoid: Use \. when matching a literal dot character.
Mistake: Python interpreting backslashes in complex patterns. Avoid: Prefix the pattern string with r (raw string).
Mistake: Using re.search when replacement is needed. Avoid: Use re.sub for string modification and replacement tasks.

FAQs

Are regex symbols the same in all programming languages? Yes, the core symbols and syntax for regular expressions are universal across almost all programming languages.
How can I test my regex pattern before writing Python code? Use the find/search function (Ctrl+F) in your text editor and enable the regular expression search mode (often Alt+R).
What is the purpose of using parentheses () in a pattern? Parentheses create capturing groups, allowing you to extract specific parts of the match or reference them later in a replacement string using backreferences (\1, \2).

Common pitfalls

Mistake: Forgetting to import the `re` package before using functions. Avoid: Always start your script with `import re`.
Mistake: Not escaping special characters like the period (`.`). Avoid: Use `\.` when matching a literal dot character.
Mistake: Python interpreting backslashes in complex patterns. Avoid: Prefix the pattern string with `r` (raw string).
Mistake: Using `re.search` when replacement is needed. Avoid: Use `re.sub` for string modification and replacement tasks.

All right, today we're going to dive into a programming fundamental, regular expressions. Regular expressions, or commonly known as regex, are a powerful tool to find patterns in text. They're basically the equivalent of using control F on a word document, except they're much more precise and powerful. This can come in handy while editing code, verifying user input, or scraping the web. The list really goes on, and that's why I want to give you a solid intro to regex today in this video. Regex are a challenging tool to master, but this tool will get you started and oriented. Today we'll go through a quick tutorial on how to validate emails and replace a string using regex in Python. Regular expressions and their symbols are universal across all programming languages. There are many symbols that can be used to find patterns in text. I'll include a link to a regex cheat sheet in the description below, as well as to the code in today's tutorial. Let's see how a few of these regular expressions work in real time. When I press control F, I get the find word prompt. If I press alt R, I can now search using regular expressions. Using control F like this is a good way to make sure your regex patterns work before you try to implement them in your Python script. When I type backslash lowercase D, I'm searching for all valid digits, so any number between 0 and 9. You can see that everywhere a number occurs, it's highlighted. Conversely, when I type backslash capital D, I'm now looking for the opposite, or non-digits. Now you can see everything but the numbers are highlighted. It's time to jump to our first example, verifying a user's email. Using regex to verify a user has entered an email correctly is commonplace. This is a very useful application because if a website has a sign-in or sign-up page, then it's extremely important to make sure that the user is inputting a valid email. The regular expressions package in Python is called RE, so let's import RE. With the package imported, let's write the regular expression pattern we are looking for. Since email accounts follow a pattern, we can use a combination of regex symbols to only accept correctly formatted emails. We know that the first part of an email can consist of uppercase letters, lowercase letters, and numbers. So, we can specify that as such using a regex syntax. The brackets allow us to specify that we are looking for the characters in a given group. This group starts at the beginning of the word and will go until we encounter an at symbol. The plus sign after the brackets means we are looking for any combinations of one or more of these characters. Since email addresses are issued from many domains, we'll specify that we are looking for any combination of one or more upper and lowercase letters. All that's left is verifying that there is a top-level domain. We can look for a dot by adding backslash and period to the pattern. Similar to domain names, we'll just look for the presence of valid characters since there are so many top-level domains used in email addresses. Let's just account for some of the most popular ones in this example, like .com, .edu, and .net. Just add the following to the pattern so it makes sure the email ends in a .com or .edu or .net. Great. Now, let's make sure this works with an if statement and the search function from RE. First, let's get a user's input. Let's see if a string from our specified pattern is found in the user's input by calling re.search. We can use the pattern argument for the pattern we just wrote and the string argument for the user input. You can scroll down in the Kite Co-pilot for example implementations of regex patterns. So, if the pattern is found in the user input, we will print out valid email. Otherwise, we'll just print out invalid email. Good. Now, let's test our script. Let's input a correct email. Great. It correctly verified that email. Now, let's give the script an invalid email. Perfect. It detected that the email is missing the period. I wanted to take a moment to tell you more about Kite, which is an AI coding assistant that's being used in this video. Kite is a free plug-in for your code editor that uses machine learning to save you keystrokes while you're programming. If you're using Atom, VS Code, Spyder, PyCharm, Sublime, or Vim, Kite will seamlessly integrate into your coding workflow. Kite can complete entire lines of code and has a feature called intelligent snippets that will help you fill in arguments and method calls with variables defined earlier in the script. The window you'll see on the right side of my screen throughout the video is also a Kite feature called the Kite co-pilot. It automatically shows you relevant Python documentation while you type based on your cursor location. The best part of Kite is that it's free and you can download it from the link in the description below. Let's look at one more application of regular expressions. In this example, we will be looking at how regular expressions can be used to quickly replace a specific part of a string. This is extremely useful when making changes to a large database where you may have thousands of strings you need to change. Let's pretend that we need every phone number entered into a continuous string of numbers, no hyphens. But, we want to keep the hyphens that are in words. Like before, we must first write the regular expression pattern we are looking for and test it. Phone numbers come in a predictable format, so we can write a regular expression to identify them in our database. Each set of parentheses resembles a group. So, from left to right, we have groups 1, 2, and 3. But now we need to write what we want this pattern to turn into. Let's preserve the groups but remove the hyphens. Each backslash number represents a group. So, our new pattern is concatenating the three groups together without the hyphen. We put the R before the string in order for it to be interpreted as a raw string. Let's just take input from the user like we did in the previous example. And then let's use re.sub to replace the hyphens with white space. As seen in the Kite co-pilot here, we're using our pattern variable for the argument pattern and our new pattern variable for the argument repl and our user input variable for the argument string. And let's make sure we print out our result at the end. Let's try this out. There we go. You can see that all hyphens were removed from the phone number. So, that was just a sample of what regular expressions can do. This is a vital skill to learn in computer science, not only because of its usefulness, but also because it shows up in almost every language. Thanks for watching this video on regular expressions in Python. Make sure to subscribe to our channel as we'll have more Python tips coming your way. And finally, don't forget to check out the Kite AI auto-complete plugin. The link's in the description below. See you next time.

Frequently asked questions

Are regex symbols the same in all programming languages?

Yes, the core symbols and syntax for regular expressions are universal across almost all programming languages.

How can I test my regex pattern before writing Python code?

Use the find/search function (Ctrl+F) in your text editor and enable the regular expression search mode (often Alt+R).

What is the purpose of using parentheses () in a pattern?

Parentheses create capturing groups, allowing you to extract specific parts of the match or reference them later in a replacement string using backreferences (\1, \2).

How was this lesson?

Your feedback helps us refine explanations and catch bugs.