- publishing free software manuals
Perl Language Reference Manual
by Larry Wall and others
Paperback (6"x9"), 724 pages
ISBN 9781906966027
RRP £29.95 ($39.95)

Sales of this book support The Perl Foundation! Get a printed copy>>>

11.3 Extended Patterns

Perl also defines a consistent extension syntax for features not found in standard tools like awk and lex. The syntax is a pair of parentheses with a question mark as the first thing within the parentheses. The character after the question mark indicates the extension.

The stability of these extensions varies widely. Some have been part of the core language for many years. Others are experimental and may change without warning or be completely removed. Check the documentation on an individual feature to verify its current status.

A question mark was chosen for this and for the minimal-matching construct because 1) question marks are rare in older regular expressions, and 2) whenever you see one, you should stop and "question" exactly what is going on. That's psychology...

(?#text)
A comment. The text is ignored. If the /x modifier enables whitespace formatting, a simple # will suffice. Note that Perl closes the comment as soon as it sees a ), so there is no way to put a literal ) in the comment.
(?pimsx-imsx)
One or more embedded pattern-match modifiers, to be turned on (or turned off, if preceded by -) for the remainder of the pattern or the remainder of the enclosing pattern group (if any). This is particularly useful for dynamic patterns, such as those read in from a configuration file, taken from an argument, or specified in a table somewhere. Consider the case where some patterns want to be case sensitive and some do not: The case insensitive ones merely need to include (?i) at the front of the pattern. For example:
$pattern = "foobar";
if ( /$pattern/i ) { }
# more flexible:
$pattern = "(?i)foobar";
if ( /$pattern/ ) { }
These modifiers are restored at the end of the enclosing group. For example,
( (?i) blah ) \s+ \1
will match blah in any case, some spaces, and an exact (including the case!) repetition of the previous word, assuming the /x modifier, and no /i modifier outside this group. These modifiers do not carry over into named subpatterns called in the enclosing group. In other words, a pattern such as ((?i)(&NAME)) does not change the case-sensitivity of the "NAME" pattern. Note that the p modifier is special in that it can only be enabled, not disabled, and that its presence anywhere in a pattern has a global effect. Thus (?-p) and (?-p:...) are meaningless and will warn when executed under use warnings.
(?:pattern)
(?imsx-imsx:pattern)
This is for clustering, not capturing; it groups subexpressions like "()", but doesn't make backreferences as "()" does. So
@fields = split(/\b(?:a|b|c)\b/)
is like
@fields = split(/\b(a|b|c)\b/)
but doesn't spit out extra fields. It's also cheaper not to capture characters if you don't need to. Any letters between ? and : act as flags modifiers as with (?imsx-imsx). For example,
/(?s-i:more.*than).*million/i
is equivalent to the more verbose
/(?:(?s-i)more.*than).*million/i
(?|pattern)
This is the "branch reset" pattern, which has the special property that the capture buffers are numbered from the same starting point in each alternation branch. It is available starting from perl 5.10.0. Capture buffers are numbered from left to right, but inside this construct the numbering is restarted for each branch. The numbering within each branch will be as normal, and any buffers following this construct will be numbered as though the construct contained only one branch, that being the one with the most capture buffers in it. This construct will be useful when you want to capture one of a number of alternative matches. Consider the following pattern. The numbers underneath show in which buffer the captured content will be stored.
# before  ---------------branch-reset----------- after        
/ ( a )  (?| x ( y ) z | (p (q) r) | (t) u (v) ) ( z ) /x
# 1            2         2  3        2     3     4
Be careful when using the branch reset pattern in combination with named captures. Named captures are implemented as being aliases to numbered buffers holding the captures, and that interferes with the implementation of the branch reset pattern. If you are using named captures in a branch reset pattern, it's best to use the same names, in the same order, in each of the alternations:
/(?|  (?<a> x ) (?<b> y )
   |  (?<a> z ) (?<b> w )) /x
Not doing so may lead to surprises:
"12" =~ /(?| (?<a> \d+ ) | (?<b> \D+))/x;
say $+ {a};   # Prints '12'
say $+ {b};   # *Also* prints '12'.
The problem here is that both the buffer named a and the buffer named b are aliases for the buffer belonging to $1.
Look-Around Assertions
Look-around assertions are zero width patterns which match a specific pattern without including it in $&. Positive assertions match when their subpattern matches, negative assertions match when their subpattern fails. Look-behind matches text up to the current match position, look-ahead matches text following the current match position.
(?=pattern)
A zero-width positive look-ahead assertion. For example, /\w+(?=\t)/ matches a word followed by a tab, without including the tab in $&.
(?!pattern)
A zero-width negative look-ahead assertion. For example /foo(?!bar)/ matches any occurrence of "foo" that isn't followed by "bar". Note however that look-ahead and look-behind are NOT the same thing. You cannot use this for look-behind. If you are looking for a "bar" that isn't preceded by a "foo", /(?!foo)bar/ will not do what you want. That's because the (?!foo) is just saying that the next thing cannot be "foo"--and it's not, it's a "bar", so "foobar" will match. You would have to do something like /(?!foo)...bar/ for that. We say "like" because there's the case of your "bar" not having three characters before it. You could cover that this way: /(?:(?!foo)...|^.{0,2})bar/. Sometimes it's still easier just to say:
if (/bar/ && $` !~ /foo$/)
For look-behind see below.
(?<=pattern) \K
A zero-width positive look-behind assertion. For example, /(?<=\t)\w+/ matches a word that follows a tab, without including the tab in $&. Works only for fixed-width look-behind. There is a special form of this construct, called \K, which causes the regex engine to "keep" everything it had matched prior to the \K and not include it in $&. This effectively provides variable length look-behind. The use of \K inside of another look-around assertion is allowed, but the behaviour is currently not well defined. For various reasons \K may be significantly more efficient than the equivalent (?<=...) construct, and it is especially useful in situations where you want to efficiently remove something following something else in a string. For instance
s/(foo)bar/$1/g;
can be rewritten as the much more efficient
s/foo\Kbar//g;
(?<!pattern)
A zero-width negative look-behind assertion. For example /(?<!bar)foo/ matches any occurrence of "foo" that does not follow "bar". Works only for fixed-width look-behind.
(?'NAME'pattern)
(?<NAME>pattern)
A named capture buffer. Identical in every respect to normal capturing parentheses () but for the additional fact that %+ or %- may be used after a successful match to refer to a named buffer. See perlvar for more details on the %+ and %- hashes. If multiple distinct capture buffers have the same name then the $+{NAME} will refer to the leftmost defined buffer in the match. The forms (?'NAME'pattern) and (?<NAME>pattern) are equivalent. NOTE: While the notation of this construct is the same as the similar function in .NET regexes, the behavior is not. In Perl the buffers are numbered sequentially regardless of being named or not. Thus in the pattern
/(x)(?<foo>y)(z)/
$+{foo} will be the same as $2, and $3 will contain 'z' instead of the opposite which is what a .NET regex hacker might expect. Currently NAME is restricted to simple identifiers only. In other words, it must match /^[_A-Za-z][_A-Za-z0-9]*\z/ or its Unicode extension (see "Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code" (utf8) in the Perl Library Reference Manual (Volume 1)), though it isn't extended by the locale (see "Perl locale handling (internationalization and localization)" (perllocale) in the Perl Unicode and Locales Manual). NOTE: In order to make things easier for programmers with experience with the Python or PCRE regex engines, the pattern (?P<NAME>pattern) may be used instead of (?<NAME>pattern); however this form does not support the use of single quotes as a delimiter for the name.
\k<NAME>
\k'NAME'
Named backreference. Similar to numeric backreferences, except that the group is designated by name and not number. If multiple groups have the same name then it refers to the leftmost defined group in the current match. It is an error to refer to a name not defined by a (?<NAME>) earlier in the pattern. Both forms are equivalent. NOTE: In order to make things easier for programmers with experience with the Python or PCRE regex engines, the pattern (?P=NAME) may be used instead of \k<NAME>.
(?{ code })
WARNING: This extended regular expression feature is considered experimental, and may be changed without notice. Code executed that has side effects may not perform identically from version to version due to the effect of future optimisations in the regex engine. This zero-width assertion evaluates any embedded Perl code. It always succeeds, and its code is not interpolated. Currently, the rules to determine where the code ends are somewhat convoluted. This feature can be used together with the special variable $^N to capture the results of submatches in variables without having to keep track of the number of nested parentheses. For example:
$_ = "The brown fox jumps over the lazy dog";
/the (\S+)(?{ $color = $^N }) (\S+)(?{ $animal = $^N })/i;
print "color = $color, animal = $animal\n";
Inside the (?{...}) block, $_ refers to the string the regular expression is matching against. You can also use pos() to know what is the current position of matching within this string. The code is properly scoped in the following sense: If the assertion is backtracked (compare 11.5), all changes introduced after localization are undone, so that
$_ = 'a' x 8;
m<
   (?{ $cnt = 0 })               # Initialize $cnt.
   (
     a
     (?{
         local $cnt = $cnt + 1;  # Update $cnt, backtracking-safe.
     })
   )*
   aaaa
   (?{ $res = $cnt })         # On success copy to non-localized
                              # location.
 >x;
will set $res = 4. Note that after the match, $cnt returns to the globally introduced value, because the scopes that restrict local operators are unwound. This assertion may be used as a (?(condition)yes-pattern|no-pattern) switch. If not used in this way, the result of evaluation of code is put into the special variable $^R. This happens immediately, so $^R can be used from other (?{ code }) assertions inside the same regular expression. The assignment to $^R above is properly localized, so the old value of $^R is restored if the assertion is backtracked; compare 11.5. For reasons of security, this construct is forbidden if the regular expression involves run-time interpolation of variables, unless the perilous use re 'eval' pragma has been used (see "Perl pragma to alter regular expression behaviour" (re) in the Perl Library Reference Manual (Volume 1)), or the variables contain results of qr// operator (see qr/STRING/imosx in perlop). This restriction is due to the wide-spread and remarkably convenient custom of using run-time determined strings as patterns. For example:
$re = <>;
chomp $re;
$string =~ /$re/;
Before Perl knew how to execute interpolated code within a pattern, this operation was completely safe from a security point of view, although it could raise an exception from an illegal pattern. If you turn on the use re 'eval', though, it is no longer secure, so you should only do so if you are also using taint checking. Better yet, use the carefully constrained evaluation within a Safe compartment. See 22 for details about both these mechanisms. WARNING: Use of lexical (my) variables in these blocks is broken. The result is unpredictable and will make perl unstable. The workaround is to use global (our) variables. WARNING: Because Perl's regex engine is currently not re-entrant, interpolated code may not invoke the regex engine either directly with m// or s///), or indirectly with functions such as split. Invoking the regex engine in these blocks will make perl unstable.
(??{ code })
WARNING: This extended regular expression feature is considered experimental, and may be changed without notice. Code executed that has side effects may not perform identically from version to version due to the effect of future optimisations in the regex engine. This is a "postponed" regular subexpression. The code is evaluated at run time, at the moment this subexpression may match. The result of evaluation is considered as a regular expression and matched as if it were inserted instead of this construct. Note that this means that the contents of capture buffers defined inside an eval'ed pattern are not available outside of the pattern, and vice versa, there is no way for the inner pattern to refer to a capture buffer defined outside. Thus,
('a' x 100)=~/(??{'(.)' x 100})/
will match, it will not set $1. The code is not interpolated. As before, the rules to determine where the code ends are currently somewhat convoluted. The following pattern matches a parenthesized group:
$re = qr{
           \(
           (?:
              (?> [^()]+ )    # Non-parens without backtracking
            |
              (??{ $re })     # Group with matching parens
           )*
           \)
        }x;
See also (?PARNO) for a different, more efficient way to accomplish the same task. For reasons of security, this construct is forbidden if the regular expression involves run-time interpolation of variables, unless the perilous use re 'eval' pragma has been used (see "Perl pragma to alter regular expression behaviour" (re) in the Perl Library Reference Manual (Volume 1)), or the variables contain results of qr// operator (see qr/STRING/imosx in perlop). Because perl's regex engine is not currently re-entrant, delayed code may not invoke the regex engine either directly with m// or s///), or indirectly with functions such as split. Recursing deeper than 50 times without consuming any input string will result in a fatal error. The maximum depth is compiled into perl, so changing it requires a custom build.
(?PARNO) (?-PARNO) (?+PARNO) (?R) (?0)
Similar to (??{ code }) except it does not involve compiling any code, instead it treats the contents of a capture buffer as an independent pattern that must match at the current position. Capture buffers contained by the pattern will have the value as determined by the outermost recursion. PARNO is a sequence of digits (not starting with 0) whose value reflects the paren-number of the capture buffer to recurse to. (?R) recurses to the beginning of the whole pattern. (?0) is an alternate syntax for (?R). If PARNO is preceded by a plus or minus sign then it is assumed to be relative, with negative numbers indicating preceding capture buffers and positive ones following. Thus (?-1) refers to the most recently declared buffer, and (?+1) indicates the next buffer to be declared. Note that the counting for relative recursion differs from that of relative backreferences, in that with recursion unclosed buffers are included. The following pattern matches a function foo() which may contain balanced parentheses as the argument.
$re = qr{ (                    # paren group 1 (full function)
            foo
            (                  # paren group 2 (parens)
              \(
                (              # paren group 3 (contents of parens)
                (?:
                 (?> [^()]+ )  # Non-parens without backtracking
                |
                 (?2)          # Recurse to start of paren group 2
                )*
                )
              \)
            )
          )
        }x;
If the pattern was used as follows
'foo(bar(baz)+baz(bop))'=~/$re/
    and print "\$1 = $1\n",
              "\$2 = $2\n",
              "\$3 = $3\n";
the output produced should be the following:
$1 = foo(bar(baz)+baz(bop))
$2 = (bar(baz)+baz(bop))
$3 = bar(baz)+baz(bop)
If there is no corresponding capture buffer defined, then it is a fatal error. Recursing deeper than 50 times without consuming any input string will also result in a fatal error. The maximum depth is compiled into perl, so changing it requires a custom build. The following shows how using negative indexing can make it easier to embed recursive patterns inside of a qr// construct for later use:
my $parens = qr/(\((?:[^()]++|(?-1))*+\))/;
if (/foo $parens \s+ + \s+ bar $parens/x) {
   # do something here...
}
Note that this pattern does not behave the same way as the equivalent PCRE or Python construct of the same form. In Perl you can backtrack into a recursed group, in PCRE and Python the recursed into group is treated as atomic. Also, modifiers are resolved at compile time, so constructs like (?i:(?1)) or (?:(?i)(?1)) do not affect how the sub-pattern will be processed.
(?&NAME)
Recurse to a named subpattern. Identical to (?PARNO) except that the parenthesis to recurse to is determined by name. If multiple parentheses have the same name, then it recurses to the leftmost. It is an error to refer to a name that is not declared somewhere in the pattern. NOTE: In order to make things easier for programmers with experience with the Python or PCRE regex engines the pattern (?P>NAME) may be used instead of (?&NAME).
(?(condition)yes-pattern|no-pattern)
(?(condition)yes-pattern)
Conditional expression. (condition) should be either an integer in parentheses (which is valid if the corresponding pair of parentheses matched), a look-ahead/look-behind/evaluate zero-width assertion, a name in angle brackets or single quotes (which is valid if a buffer with the given name matched), or the special symbol (R) (true when evaluated inside of recursion or eval). Additionally the R may be followed by a number, (which will be true when evaluated when recursing inside of the appropriate group), or by &NAME, in which case it will be true only when evaluated during recursion in the named group. Here's a summary of the possible predicates:
(1) (2) ...
Checks if the numbered capturing buffer has matched something.
(<NAME>) ('NAME')
Checks if a buffer with the given name has matched something.
(?{ CODE })
Treats the code block as the condition.
(R)
Checks if the expression has been evaluated inside of recursion.
(R1) (R2) ...
Checks if the expression has been evaluated while executing directly inside of the n-th capture group. This check is the regex equivalent of
if ((caller(0))[3] eq 'subname') { ... }
In other words, it does not check the full recursion stack.
(R&NAME)
Similar to (R1), this predicate checks to see if we're executing directly inside of the leftmost group with a given name (this is the same logic used by (?&NAME) to disambiguate). It does not check the full stack, but only the name of the innermost active recursion.
(DEFINE)
In this case, the yes-pattern is never directly executed, and no no-pattern is allowed. Similar in spirit to (?{0}) but more efficient. See below for details.
For example:
m{ ( \( )?
   [^()]+
   (?(1) \) )
 }x
matches a chunk of non-parentheses, possibly included in parentheses themselves. A special form is the (DEFINE) predicate, which never executes directly its yes-pattern, and does not allow a no-pattern. This allows to define subpatterns which will be executed only by using the recursion mechanism. This way, you can define a set of regular expression rules that can be bundled into any pattern you choose. It is recommended that for this usage you put the DEFINE block at the end of the pattern, and that you name any subpatterns defined within it. Also, it's worth noting that patterns defined this way probably will not be as efficient, as the optimiser is not very clever about handling them. An example of how this might be used is as follows:
/(?<NAME>(?&NAME_PAT))(?<ADDR>(?&ADDRESS_PAT))
 (?(DEFINE)
   (?<NAME_PAT>....)
   (?<ADRESS_PAT>....)
 )/x
Note that capture buffers matched inside of recursion are not accessible after the recursion returns, so the extra layer of capturing buffers is necessary. Thus $+{NAME_PAT} would not be defined even though $+{NAME} would be.
(?>pattern)
An "independent" subexpression, one which matches the substring that a standalone pattern would match if anchored at the given position, and it matches nothing other than this substring. This construct is useful for optimizations of what would otherwise be "eternal" matches, because it will not backtrack (see 11.5). It may also be useful in places where the "grab all you can, and do not give anything back" semantic is desirable. For example: ^(?>a*)ab will never match, since (?>a*) (anchored at the beginning of string, as above) will match all characters a at the beginning of string, leaving no a for ab to match. In contrast, a*ab will match the same as a+b, since the match of the subgroup a* is influenced by the following group ab (see 11.5). In particular, a* inside a*ab will match fewer characters than a standalone a*, since this makes the tail match. An effect similar to (?>pattern) may be also achieved with the expression (?=(pattern))\1. This matches the same substring as a standalone a+, and the following \1 eats the matched string; it therefore makes a zero-length assertion into an analogue of (?>...). (The difference between these two constructs is that the second one uses a capturing group, thus shifting ordinals of backreferences in the rest of a regular expression.) Consider this pattern:
m{ \(
      (
        [^()]+              # x+
      |
        \( [^()]* \)
      )+
   \)
 }x
That will efficiently match a nonempty group with matching parentheses two levels deep or less. However, if there is no such group, it will take virtually forever on a long string. That's because there are so many different ways to split a long string into several substrings. This is what (.+)+ is doing, and (.+)+ is similar to a subpattern of the above pattern. Consider how the pattern above detects no-match on ((()aaaaaaaaaaaaaaaaaa in several seconds, but that each extra letter doubles this time. This exponential performance will make it appear that your program has hung. However, a tiny change to this pattern
m{ \(
      (
        (?> [^()]+ )        # change x+ above to (?> x+ )
      |
        \( [^()]* \)
      )+
   \)
 }x
which uses (?>...) matches exactly when the one above does (verifying this yourself would be a productive exercise), but finishes in a fourth the time when used on a similar string with 1000000 as. Be aware, however, that this pattern currently triggers a warning message under the use warnings pragma or -w switch saying it "matches null string many times in regex". On simple groups, such as the pattern (?> [^()]+ ), a comparable effect may be achieved by negative look-ahead, as in [^()]+ (?! [^()] ). This was only 4 times slower on a string with 1000000 as. The "grab all you can, and do not give anything back" semantic is desirable in many situations where on the first sight a simple ()* looks like the correct solution. Suppose we parse text with comments being delimited by # followed by some optional (horizontal) whitespace. Contrary to its appearance, #[ \t]* is not the correct subexpression to match the comment delimiter, because it may "give up" some whitespace if the remainder of the pattern can be made to match that way. The correct answer is either one of these:
(?>#[ \t]*)
#[ \t]*(?![ \t])
For example, to grab non-empty comments into $1, one should use either one of these:
/ (?> \# [ \t]* ) (        .+ ) /x;
/     \# [ \t]*   ( [^ \t] .* ) /x;
Which one you pick depends on which of these expressions better reflects the above specification of comments. In some literature this construct is called "atomic matching" or "possessive matching". Possessive quantifiers are equivalent to putting the item they are applied to inside of one of these constructs. The following equivalences apply:
Quantifier Form Bracketing Form
PAT*+ (?>PAT*)
PAT++ (?>PAT+)
PAT?+ (?>PAT?)
PAT{min,max}+ (?>PAT{min,max})
ISBN 9781906966027Perl Language Reference ManualSee the print edition