- •Introduction
- •Who This Book Is For
- •What This Book Covers
- •How This Book Is Structured
- •What You Need to Use This Book
- •Conventions
- •Source Code
- •Errata
- •p2p.wrox.com
- •What Are Regular Expressions?
- •What Can Regular Expressions Be Used For?
- •Finding Doubled Words
- •Checking Input from Web Forms
- •Changing Date Formats
- •Finding Incorrect Case
- •Adding Links to URLs
- •Regular Expressions You Already Use
- •Search and Replace in Word Processors
- •Directory Listings
- •Online Searching
- •Why Regular Expressions Seem Intimidating
- •Compact, Cryptic Syntax
- •Whitespace Can Significantly Alter the Meaning
- •No Standards Body
- •Differences between Implementations
- •Characters Change Meaning in Different Contexts
- •Regular Expressions Can Be Case Sensitive
- •Case-Sensitive and Case-Insensitive Matching
- •Case and Metacharacters
- •Continual Evolution in Techniques Supported
- •Multiple Solutions for a Single Problem
- •What You Want to Do with a Regular Expression
- •Replacing Text in Quantity
- •Regular Expression Tools
- •findstr
- •Microsoft Word
- •StarOffice Writer/OpenOffice.org Writer
- •Komodo Rx Package
- •PowerGrep
- •Microsoft Excel
- •JavaScript and JScript
- •VBScript
- •Visual Basic.NET
- •Java
- •Perl
- •MySQL
- •SQL Server 2000
- •W3C XML Schema
- •An Analytical Approach to Using Regular Expressions
- •Express and Document What You Want to Do in English
- •Consider the Regular Expression Options Available
- •Consider Sensitivity and Specificity
- •Create Appropriate Regular Expressions
- •Document All but Simple Regular Expressions
- •Document What You Expect the Regular Expression to Do
- •Document What You Want to Match
- •Test the Results of a Regular Expression
- •Matching Single Characters
- •Matching Sequences of Characters That Each Occur Once
- •Introducing Metacharacters
- •Matching Sequences of Different Characters
- •Matching Optional Characters
- •Matching Multiple Optional Characters
- •Other Cardinality Operators
- •The * Quantifier
- •The + Quantifier
- •The Curly-Brace Syntax
- •The {n} Syntax
- •The {n,m} Syntax
- •Exercises
- •Regular Expression Metacharacters
- •Thinking about Characters and Positions
- •The Period (.) Metacharacter
- •Matching Variably Structured Part Numbers
- •Matching a Literal Period
- •The \w Metacharacter
- •The \W Metacharacter
- •Digits and Nondigits
- •The \d Metacharacter
- •Canadian Postal Code Example
- •The \D Metacharacter
- •Alternatives to \d and \D
- •The \s Metacharacter
- •Handling Optional Whitespace
- •The \S Metacharacter
- •The \t Metacharacter
- •The \n Metacharacter
- •Escaped Characters
- •Finding the Backslash
- •Modifiers
- •Global Search
- •Case-Insensitive Search
- •Exercises
- •Introduction to Character Classes
- •Choice between Two Characters
- •Using Quantifiers with Character Classes
- •Using the \b Metacharacter in Character Classes
- •Selecting Literal Square Brackets
- •Using Ranges in Character Classes
- •Alphabetic Ranges
- •Use [A-z] With Care
- •Digit Ranges in Character Classes
- •Hexadecimal Numbers
- •IP Addresses
- •Reverse Ranges in Character Classes
- •A Potential Range Trap
- •Finding HTML Heading Elements
- •Metacharacter Meaning within Character Classes
- •The ^ metacharacter
- •How to Use the - Metacharacter
- •Negated Character Classes
- •Combining Positive and Negative Character Classes
- •POSIX Character Classes
- •The [:alnum:] Character Class
- •Exercises
- •String, Line, and Word Boundaries
- •The ^ Metacharacter
- •The ^ Metacharacter and Multiline Mode
- •The $ Metacharacter
- •The $ Metacharacter in Multiline Mode
- •Using the ^ and $ Metacharacters Together
- •Matching Blank Lines
- •Working with Dollar Amounts
- •Revisiting the IP Address Example
- •What Is a Word?
- •Identifying Word Boundaries
- •The \< Syntax
- •The \>Syntax
- •The \b Syntax
- •The \B Metacharacter
- •Less-Common Word-Boundary Metacharacters
- •Exercises
- •Grouping Using Parentheses
- •Parentheses and Quantifiers
- •Matching Literal Parentheses
- •U.S. Telephone Number Example
- •Alternation
- •Choosing among Multiple Options
- •Unexpected Alternation Behavior
- •Capturing Parentheses
- •Numbering of Captured Groups
- •Numbering When Using Nested Parentheses
- •Named Groups
- •Non-Capturing Parentheses
- •Back References
- •Exercises
- •Why You Need Lookahead and Lookbehind
- •The (? metacharacters
- •Lookahead
- •Positive Lookahead
- •Negative Lookahead
- •Positive Lookahead Examples
- •Positive Lookahead in the Same Document
- •Inserting an Apostrophe
- •Lookbehind
- •Positive Lookbehind
- •Negative Lookbehind
- •How to Match Positions
- •Adding Commas to Large Numbers
- •Exercises
- •What Are Sensitivity and Specificity?
- •Extreme Sensitivity, Awful Specificity
- •Email Addresses Example
- •Replacing Hyphens Example
- •The Sensitivity/Specificity Trade-Off
- •Sensitivity, Specificity, and Positional Characters
- •Sensitivity, Specificity, and Modes
- •Sensitivity, Specificity, and Lookahead and Lookbehind
- •How Much Should the Regular Expressions Do?
- •Abbreviations
- •Characters from Other Languages
- •Names
- •Sensitivity and How to Achieve It
- •Specificity and How to Maximize It
- •Exercises
- •Documenting Regular Expressions
- •Document the Problem Definition
- •Add Comments to Your Code
- •Making Use of Extended Mode
- •Know Your Data
- •Abbreviations
- •Proper Names
- •Incorrect Spelling
- •Creating Test Cases
- •Debugging Regular Expressions
- •Treacherous Whitespace
- •Backslashes Causing Problems
- •Considering Other Causes
- •The User Interface
- •Metacharacters Available
- •Quantifiers
- •The @ Quantifier
- •The {n,m} Syntax
- •Modes
- •Character Classes
- •Back References
- •Lookahead and Lookbehind
- •Lazy Matching versus Greedy Matching
- •Examples
- •Character Class Examples, Including Ranges
- •Whole Word Searches
- •Search-and-Replace Examples
- •Changing Name Structure Using Back References
- •Manipulating Dates
- •The Star Training Company Example
- •Regular Expressions in Visual Basic for Applications
- •Exercises
- •The User Interface
- •Metacharacters Available
- •Quantifiers
- •Modes
- •Character Classes
- •Alternation
- •Back References
- •Lookahead and Lookbehind
- •Search Example
- •Search-and-Replace Example
- •Online Chats
- •POSIX Character Classes
- •Matching Numeric Digits
- •Exercises
- •Introducing findstr
- •Finding Literal Text
- •Quantifiers
- •Character Classes
- •Command-Line Switch Examples
- •The /v Switch
- •The /a Switch
- •Single File Examples
- •Simple Character Class Example
- •Find Protocols Example
- •Multiple File Example
- •A Filelist Example
- •Exercises
- •The PowerGREP Interface
- •A Simple Find Example
- •The Replace Tab
- •The File Finder Tab
- •Syntax Coloring
- •Other Tabs
- •Numeric Digits and Alphabetic Characters
- •Quantifiers
- •Back References
- •Alternation
- •Line Position Metacharacters
- •Word-Boundary Metacharacters
- •Lookahead and Lookbehind
- •Longer Examples
- •Finding HTML Horizontal Rule Elements
- •Matching Time Example
- •Exercises
- •The Excel Find Interface
- •Escaping Wildcard Characters
- •Using Wildcards in Data Forms
- •Using Wildcards in Filters
- •Exercises
- •Using LIKE with Regular Expressions
- •The % Metacharacter
- •The _ Metacharacter
- •Character Classes
- •Negated Character Classes
- •Using Full-Text Search
- •Using The CONTAINS Predicate
- •Document Filters on Image Columns
- •Exercises
- •Using the _ and % Metacharacters
- •Testing Matching of Literals: _ and % Metacharacters
- •Using Positional Metacharacters
- •Using Character Classes
- •Quantifiers
- •Social Security Number Example
- •Exercises
- •The Interface to Metacharacters in Microsoft Access
- •Creating a Hard-Wired Query
- •Creating a Parameter Query
- •Using the ? Metacharacter
- •Using the * Metacharacter
- •Using the # Metacharacter
- •Using the # Character with Date/Time Data
- •Using Character Classes in Access
- •Exercises
- •The RegExp Object
- •Attributes of the RegExp Object
- •The Other Properties of the RegExp Object
- •The test() Method of the RegExp Object
- •The exec() Method of the RegExp Object
- •The String Object
- •Metacharacters in JavaScript and JScript
- •SSN Validation Example
- •Exercises
- •The RegExp Object and How to Use It
- •Quantifiers
- •Positional Metacharacters
- •Character Classes
- •Word Boundaries
- •Lookahead
- •Grouping and Nongrouping Parentheses
- •Exercises
- •The System.Text.RegularExpressions namespace
- •A Simple Visual Basic .NET Example
- •The Classes of System.Text.RegularExpressions
- •The Regex Object
- •Using the Match Object and Matches Collection
- •Using the Match.Success Property and Match.NextMatch Method
- •The GroupCollection and Group Classes
- •The CaptureCollection and Capture Class
- •The RegexOptions Enumeration
- •Case-Insensitive Matching: The IgnoreCase Option
- •Multiline Matching: The Effect on the ^ and $ Metacharacters
- •Right to Left Matching: The RightToLeft Option
- •Lookahead and Lookbehind
- •Exercises
- •An Introductory Example
- •The Classes of System.Text.RegularExpressions
- •The Regex Class
- •The Options Property of the Regex Class
- •Regex Class Methods
- •The CompileToAssembly() Method
- •The GetGroupNames() Method
- •The GetGroupNumbers() Method
- •GroupNumberFromName() and GroupNameFromNumber() Methods
- •The IsMatch() Method
- •The Match() Method
- •The Matches() Method
- •The Replace() Method
- •The Split() Method
- •Using the Static Methods of the Regex Class
- •The IsMatch() Method as a Static
- •The Match() Method as a Static
- •The Matches() Method as a Static
- •The Replace() Method as a Static
- •The Split() Method as a Static
- •The Match and Matches Classes
- •The Match Class
- •The GroupCollection and Group Classes
- •The RegexOptions Class
- •The IgnorePatternWhitespace Option
- •Metacharacters Supported in Visual C# .NET
- •Using Named Groups
- •Using Back References
- •Exercise
- •The ereg() Set of Functions
- •The ereg() Function
- •The ereg() Function with Three Arguments
- •The eregi() Function
- •The ereg_replace() Function
- •The eregi_replace() Function
- •The split() Function
- •The spliti() Function
- •The sql_regcase() Function
- •Perl Compatible Regular Expressions
- •Pattern Delimiters in PCRE
- •Escaping Pattern Delimiters
- •Matching Modifiers in PCRE
- •Using the preg_match() Function
- •Using the preg_match_all() Function
- •Using the preg_grep() Function
- •Using the preg_quote() Function
- •Using the preg_replace() Function
- •Using the preg_replace_callback() Function
- •Using the preg_split() Function
- •Supported Metacharacters with ereg()
- •Using POSIX Character Classes with PHP
- •Supported Metacharacters with PCRE
- •Positional Metacharacters
- •Character Classes in PHP
- •Documenting PHP Regular Expressions
- •Exercises
- •W3C XML Schema Basics
- •Tools for Using W3C XML Schema
- •Comparing XML Schema and DTDs
- •How Constraints Are Expressed in W3C XML Schema
- •W3C XML Schema Datatypes
- •Derivation by Restriction
- •Unicode and W3C XML Schema
- •Unicode Overview
- •Using Unicode Character Classes
- •Matching Decimal Numbers
- •Mixing Unicode Character Classes with Other Metacharacters
- •Unicode Character Blocks
- •Using Unicode Character Blocks
- •Metacharacters Supported in W3C XML Schema
- •Positional Metacharacters
- •Matching Numeric Digits
- •Alternation
- •Using the \w and \s Metacharacters
- •Escaping Metacharacters
- •Exercises
- •Introduction to the java.util.regex Package
- •Obtaining and Installing Java
- •The Pattern Class
- •Using the matches() Method Statically
- •Two Simple Java Examples
- •The Properties (Fields) of the Pattern Class
- •The CASE_INSENSITIVE Flag
- •Using the COMMENTS Flag
- •The DOTALL Flag
- •The MULTILINE Flag
- •The UNICODE_CASE Flag
- •The UNIX_LINES Flag
- •The Methods of the Pattern Class
- •The compile() Method
- •The flags() Method
- •The matcher() Method
- •The matches() Method
- •The pattern() Method
- •The split() Method
- •The Matcher Class
- •The appendReplacement() Method
- •The appendTail() Method
- •The end() Method
- •The find() Method
- •The group() Method
- •The groupCount() Method
- •The lookingAt() Method
- •The matches() Method
- •The pattern() Method
- •The replaceAll() Method
- •The replaceFirst() Method
- •The reset() Method
- •The start() Method
- •The PatternSyntaxException Class
- •Using the \d Metacharacter
- •Character Classes
- •The POSIX Character Classes in the java.util.regex Package
- •Unicode Character Classes and Character Blocks
- •Using Escaped Characters
- •Using Methods of the String Class
- •Using the matches() Method
- •Using the replaceFirst() Method
- •Using the replaceAll() Method
- •Using the split() Method
- •Exercises
- •Obtaining and Installing Perl
- •Creating a Simple Perl Program
- •Basics of Perl Regular Expression Usage
- •Using the m// Operator
- •Using Other Regular Expression Delimiters
- •Matching Using Variable Substitution
- •Using the s/// Operator
- •Using s/// with the Global Modifier
- •Using s/// with the Default Variable
- •Using the split Operator
- •Using Quantifiers in Perl
- •Using Positional Metacharacters
- •Captured Groups in Perl
- •Using Back References in Perl
- •Using Alternation
- •Using Character Classes in Perl
- •Using Lookahead
- •Using Lookbehind
- •Escaping Metacharacters
- •A Simple Perl Regex Tester
- •Exercises
- •Index
Regular Expressions in Perl
The $& variable is used to display the whole match, in this case, the character sequence B9:
print “The whole match is ‘$&’, contained in the \$& variable.\n”;
The group captured by the first pair of parentheses matches the character class [A-Z]. In this case, the variable $1 holds the single character B:
print “The first captured group is ‘$1’, contained in ‘\$1’.\n”;
The group captured by the second pair of parentheses matches against the metacharacter \d. In this case the variable $2 holds the value 9:
print “The second captured group is ‘$2’, contained in ‘\$2’\n”;
Using Back References in Perl
Perl supports the use of back references, which are references to captured groups that can be used from inside the regular expression pattern.
A classic example of the use of back references is in the identification and correction of doubled words in text. The following example illustrates the use of back references for that purpose.
Try It Out |
Using Back References to Detect Doubled Words |
1.Type the following code into a text editor:
#!/usr/bin/perl -w use strict;
my $myPattern = “(\\w+)(\\s+\\1\\b)”;
my $myTestString = “Paris in the the Spring Spring.”; print “The original string was ‘$myTestString’.\n”; $myTestString =~ s/$myPattern/$1/g;
print “The captured group was: ‘$1’.\n”;
print “Any doubled word has now been removed.\n”; print “The string is now ‘$myTestString’.\n”;
2.Save the code as DoubledWord.pl.
3.Run the code, and inspect the displayed result, as shown in Figure 26-23. Notice in the original test string that two words were doubled: the and Spring. In the replacement string, both doubled words have been removed.
Figure 26-23
689
Chapter 26
How It Works
The pattern is assigned to the variable $myPattern. The metacharacters \w, \s, and \b have to be written with an extra backslash.
Note that the back reference \1 must also be written with an extra backslash:
my $myPattern = “(\\w+)(\\s+\\1\\b)”;
The test string Paris in the the Spring Spring. has two pairs of doubled words:
my $myTestString = “Paris in the the Spring Spring.”;
The original string containing the doubled words is displayed to the user:
print “The original string was ‘$myTestString’.\n”;
The back reference $1 is used with the s/// operator.
In the pattern, the component (\w+) captures the first word in $1. The remainder of the match is in $2, which is discarded.
The g modifier means that all occurrences of doubled words will be replaced:
$myTestString =~ s/$myPattern/$1/g;
Information about the first captured group, the effect of the replacement, and the result of the replacement is displayed to the user:
print “The captured group was: ‘$1’.\n”;
print “Any doubled word has now been removed.\n”; print “The string is now ‘$myTestString’.\n”;
Using Alternation
Alternation allows specific options to be matched. The pipe character, |, is used to express alternation.
Try It Out |
Using Alternation |
1.Type the following code into a text editor, and save it as Alternation.pl:
#!/usr/bin/perl -w use strict;
my $myPattern = “(Jim|Fred|Alice)”; print “Enter your first name here: \n”; my $myTestString = <STDIN>; chomp($myTestString);
if ($myTestString =~ m/$myPattern/)
{
print “Hello $&. How are you?”;
}
else
690
Regular Expressions in Perl
{
print “I am sorry, $myTestString. I don’t know you.”;
}
2.Run the code, enter the name Alice, and inspect the displayed results.
3.Run the code again; enter the name Andrew; and inspect the displayed results, as shown in Figure 26-24.
Figure 26-24
How It Works
The $myPattern variable is assigned a pattern that uses the pipe character to specify three literal patterns as options:
my $myPattern = “(Jim|Fred|Alice)”;
The user is asked to enter his or her first name:
print “Enter your first name here: \n”;
A line of characters from the standard input is assigned to the $myTestString variable:
my $myTestString = <STDIN>;
The chomp() operator is used to remove the newline character at the end of $myTestString:
chomp($myTestString);
If the name entered by the user is one of the three specified options, a message greeting the user is displayed:
if ($myTestString =~ m/$myPattern/)
{
print “Hello $&. How are you?”;
}
However, if the name entered is not one of the three permitted options, the user is told that he or she is not known:
else
{
print “I am sorry, $myTestString. I don’t know you.”;
}
691
Chapter 26
Using Character Classes in Perl
Perl supports an extensive range of character class functionality. If you want to specify individual characters to be matched, you simply list those inside a character class.
Metacharacters inside character classes are different from metacharacters outside them. Outside a character class, the ^ metacharacter matches a position before the first character of a string or line (depending on settings). Inside a character class, the ^ metacharacter, when it is the first character after the left square bracket, indicates a negated character class. All the characters after the ^ are characters that do not match.
Try It Out |
Using a Character Class |
1.Type the following code in a text editor, and save it as CharacterClass.pl:
#!/usr/bin/perl -w use strict;
print “Enter a character class to be used as a pattern: “; my $myPattern = <STDIN>;
print “\n\nEnter a string to test against the character class: “; my $myTestString = <STDIN>;
chomp ($myPattern); chomp ($myTestString);
print “\n\nThe string you entered was: ‘$myTestString’.\n”; print “The pattern you entered was: ‘$myPattern’.\n”;
if ($myTestString =~ m/$myPattern/)
{
print “There was a match: ‘$&’.\n”;
}
else
{
print “There was no match.”;
}
2.Run the code.
3.Enter the pattern [A-Z][a-z]*.
4.Enter the test string Hello world!, and inspect the displayed results.
5.Run the code again.
6.Enter the pattern [A-E][a-z]*.
7.Enter the test string Hello Ethel. How are you?, and inspect the displayed results, as shown in Figure 26-25.
692
Regular Expressions in Perl
Figure 26-25
How It Works
The user is invited to enter a pattern to be matched against:
print “Enter a character class to be used as a pattern: “;
A line of characters from the standard input is assigned to the $myPattern variable:
my $myPattern = <STDIN>;
The user is then invited to enter a test string. The test string is captured from the standard input and is assigned to the $myTestString variable:
print “\n\nEnter a string to test against the character class: “;
my $myTestString = <STDIN>;
Then the chomp() operator is used to remove the terminal newline character from $myPattern and from $myTestString:
chomp ($myPattern);
chomp ($myTestString);
The pattern and test string are displayed to ensure that the user is aware of both. The user should identify any typos from that information:
print “\n\nThe string you entered was: ‘$myTestString’.\n”;
print “The pattern you entered was: ‘$myPattern’.\n”;
An if statement uses a matching process to determine whether a message about success or failure of matching has occurred:
if ($myTestString =~ m/$myPattern/)
{
693
Chapter 26
If the match is successful, the content of the match, which is contained in the $& variable, is displayed:
print “There was a match: ‘$&’.\n”;
}
If matching is unsuccessful, the statement specified by the else clause is executed:
else
{
print “There was no match.”;
}
When the pattern is [A-Z][a-z]*, it matches an uppercase alphabetic character followed by zero or more lowercase alphabetic characters. In the test string Hello world!, the first matching character sequence is Hello. Matching is greedy, in that it matches as many characters as possible.
When the pattern is [A-E][a-z]*, the initial uppercase alphabetic character must be in the range A through E. Therefore, the H of Hello does not match. However, the E of Ethel does match against [A-E]. The E is followed by lowercase alphabetic characters, so the entire match is Ethel, as shown in Figure 26-25.
Negated character classes specify that a character class matches a character that is not one of those contained between the square brackets. The ^ metacharacter specifies that it is a negated character class if it is the first character after the opening square bracket.
Try It Out |
Using a Negated Character Class |
1.Type the following code in a text editor:
#!/usr/bin/perl -w use strict;
my $myPattern = “[^A-D]\\d{2}”;
my $myTestString = “A99 B23 C34 D45 E55”;
print “The test string is: ‘$myTestString’.\n”; print “The pattern is: ‘$myPattern’.\n”;
if ($myTestString =~ m/$myPattern/)
{
print “There was a match: ‘$&’.\n”;
}
else
{
print “There was no match.”;
}
2.
3.
Save the code as NegatedCharacterClass.pl.
Run the code, and inspect the displayed results, as shown in Figure 26-26.
694
Regular Expressions in Perl
Figure 26-26
How It Works
The pattern assigned to the $myPattern variable is [^A-D]\d{2}. Remember, it is necessary to double the backslash to ensure that the \d metacharacter is correctly recognized. The pattern [^A-D]\d{2} matches a character that is not A through D, followed by two numeric digits:
my $myPattern = “[^A-D]\\d{2}”;
The test string is assigned to the include an uppercase alphabetic will not match:
$myTestString variable. Notice that the first four character sequences character in the range A through D, which the negated character class
my $myTestString = “A99 B23 C34 D45 E55”;
The test string and pattern are displayed:
print “The test string is: ‘$myTestString’.\n”;
print “The pattern is: ‘$myPattern’.\n”;
The if statement uses a test that determines whether or not there is a match:
if ($myTestString =~ m/$myPattern/)
Because the negated character class [^A-D] won’t match an uppercase character A through D, the first match is E55. That value is, therefore, displayed using the $& variable:
print “There was a match: ‘$&’.\n”;
You saw earlier in this chapter how variable substitution can be used in other settings. Variable substitution can also be used in character classes.
Try It Out |
Using Variable Substitution in a Character Class |
1.Type the following code in a text editor:
#!/usr/bin/perl -w use strict;
my $toBeSubstituted = “A-D”;
my $myPattern = “[$toBeSubstituted]\\d{2}”; my $myTestString = “A99 B23 C34 D45 E55”;
print “The test string is: ‘$myTestString’.\n”; print “The pattern is: ‘$myPattern’.\n”;
if ($myTestString =~ m/$myPattern/)
695