16 Regular expressions
This chapter introduces regular expressions as a compact, powerful way to recognize, search, and extract patterns in text using Python’s re module. It explains the idea of “matching” and why compiling patterns with re.compile can be beneficial for performance, then shows how methods like search return a match object that tells you where and what was matched. The emphasis is on practical text-processing scenarios and why regex skills are invaluable for everyday Python work.
After covering the basics, the chapter walks through essential regex constructs: alternation (|), grouping with parentheses, character classes and ranges ([]), and repetition and optionality (+, ?, {m,n}). It highlights the importance of raw strings (r"...") to avoid double-escaping backslashes, a common source of confusion when writing patterns. A guided example builds a pattern for lines containing names and US-style phone numbers, then evolves it to extract pieces with named groups (?P
The chapter also demonstrates substitution with re.sub, including using a function as the replacement to compute context-aware results. It culminates in a lab that normalizes North American phone numbers into a consistent 1-NNN-NNN-NNNN format despite varied input styles. Readers compare approaches (targeting likely formats versus stripping to digits and reformatting), add validation rules for area and exchange codes, and consider trade-offs like precompiling patterns and runtime costs. The discussion emphasizes careful testing and warns against over-relying on AI-generated regexes without thorough validation.
Summary
- A regular expression (regex) is a way of recognizing and often extracting data from certain patterns of text.
- In Python regular expressions are handled by the re module of the standard library.
- For a complete list and explanation of the regex special characters, refer to the Python documentation.
- Adding an ‘r’ before a string tells Python to handle it as a “raw” string, and not process escape sequences.
- The most common regular expression methods are the search and sub methods.
- In addition to the search and sub methods, many other methods can be used to split strings, extract more information from match objects, look for the positions of substrings in the main argument string, and precisely control the iteration of a regex search over an argument string.
- Besides the \d special sequence, which can be used to indicate a digit character, many other special sequences are listed in the documentation.
- There are also regex flags, which you can use to control some of the more esoteric aspects of how extremely sophisticated matches are carried out.
- Regular expression methods can also be given a function to handle matches, in place of replacement expressions.
FAQ
What is a regular expression, and what does it mean to “match” text?
A regular expression (regex) is a pattern that recognizes (and often extracts) text with a particular structure. When a regex “matches” a string, it means the string contains a substring that fits the pattern.Why use re.compile, and what does a compiled regex give me?
re.compile turns a pattern string into a compiled regex object. While not strictly required, compiling improves performance when a pattern is reused and provides convenient methods like search and sub on that object.How can I match “hello” regardless of capitalization?
You can use alternation and grouping or a character class, for example:r"hello|Hello", r"(h|H)ello", or r"[hH]ello".What do square brackets [] do, and how do ranges and hyphens work?
Character classes like[...] match any one character from the set. Ranges are written as a-z or 0-9. To include a literal hyphen, place it first (e.g., [-012]). For example, [-a-zA-Z]+ matches a name-like sequence of letters and hyphens.When should I use raw strings for regexes, and why?
Use raw strings (prefixr) to prevent Python string escapes from interfering with regex escapes. For example, to match a backslash followed by “ten”, use re.compile(r"\\ten"). Without raw strings, you often need to double-escape backslashes.How do I repeat or make parts of a pattern optional?
Use quantifiers:+ (one or more), {m} (exactly m times), and ? (optional). Group with parentheses to control scope, e.g., optional US area code: r"(\d{3}-)?\d{3}-\d{4}".How do I extract specific pieces of text (like names and phone numbers)?
Capture with groups and named groups, e.g.,(?P<first>[-a-zA-Z]+). After search, use the match object’s group('first'), group('last'), etc. If an optional group didn’t match (e.g., middle name), group('middle') returns None.What does search return, and how do I check for a match?
pattern.search(text) returns None if no match, or a match object if successful. Check with an if (None is falsy). The match object records what and where was matched and supports group access.How do I replace text using a regex?
Usesub. Example: re.compile(r"the the").sub("the", text) replaces all “the the” occurrences with “the” and returns the new string. Unmatched text is left unchanged.Can I compute replacements based on each match?
Yes. Pass a function tosub. It receives the match object and returns the replacement. For instance, to convert integers to floats: pattern r"(?P<num>[0-9]+)", function returns match.group('num') + ".0", then regexp.sub(func, text).
The Quick Python Book, Fourth Edition ebook for free
