View source for Python usage notes/Regexp stuff

{{#addbodyclass:tag_tech}}
{{#addbodyclass:tag_prog}}
{{Pythonstuff}}


===Frequently used functions===

Finding patterns is usually done with one of:

* [https://docs.python.org/3/library/re.html#re.search {{inlinecode|<nowiki>re.search(pattern, string, flags=0</nowiki>)}}]
:: find anywhere within string
:: returns a match object

* [https://docs.python.org/3/library/re.html#re.match {{inlinecode|<nowiki>re.match(pattern, string, flags=0</nowiki>)}}] 
:: match at start of string (presumably slightly faster than search?) 
:: returns a match object

* [https://docs.python.org/3/library/re.html#re.findall {{inlinecode|<nowiki>re.findall(pattern, string, flags=0</nowiki>)}}] 
:: find all matches in string
:: returns a list of captured strings - shor to write when that's all you wanted

* [https://docs.python.org/3/library/re.html#re.finditer {{inlinecode|<nowiki>re.finditer(pattern, string, flags=0</nowiki>)}}] 
:: find all matches in string
:: returns a list of match objects - flexible if you wanted to do a more with each hit than just return its text

<!--
 re.search('a+b+', string, flags)

you can re.compile a string and then use that regexp_object

 re_ab.compile('a+b+', flags)
 re_ab.search(string, string, pos, endpos)


IIRC the compiled form isn't much faster (re caches patterns on first use),
so it's perhaps most useful if you need the position range part.

-->


You will often want to '''capture''' specific parts of a string, by using parentheses brackets around that part.

===Match objects===

This also relates to how to get stuff out

A '''[https://docs.python.org/3/library/re.html#match-objects match object]''' has things like:

* [https://docs.python.org/3/library/re.html#re.Match.group {{inlinecode|.group()}}] - '''string contents of each captured group'''
: note: If a single group matches multiple times, only the last match is accessible  {{comment|(you're probably doing something regexps aren't fit for)}}
: {{inlinecode|.group(1)}} is the first captured group, {{inlinecode|.group(2)}} is the second, etc. {{comment|(more technically, {{inlinecode|.group(n) is m.string[m.start(n):m.end(n)]}})}}
: there is one exception: '''.group(0)''' is not related to capturing at all, it is '''all text involved in the match''' by your regex


* [https://docs.python.org/3/library/re.html#re.Match.groups {{inlinecode|.groups()}}]
: returns a list of ''all'' explicitly captured groups's contents - all parts of the text that matched a bracketed part of your regexp


* [https://docs.python.org/3/library/re.html#re.Match.start {{inlinecode|.start()}}], [https://docs.python.org/3/library/re.html#re.Match.end 
: Note: {{inlinecode|.end()}}] {{comment|(it sometimes saves some typing to know that .span() returns the (start,end) tuple))}}
: of the whole if the argument is 0 / not given -- the default
: of a captured group if the argument is >= 1
: return -1 if the group did not contribute to the match


For an example of {{inlinecode|group()}} versus {{inlinecode|groups()}} -- say that 
<syntaxhighlight lang="python">
m = re.search(r'f(oo) ([0-9]+) ([0-9]+)', ' foo 1 2 ')
</syntaxhighlight>
Then
<syntaxhighlight lang="python">
m.groups() == ('oo', '1', '2')
</syntaxhighlight>
and, basically-equivalently,
<syntaxhighlight lang="python">
m.group(1) == 'oo'
m.group(2) == '1'
m.group(3) == '2'
</syntaxhighlight>
And the special case of "all text involved in the match"
<syntaxhighlight lang="python">
m.group(0) == 'foo 1 2'
</syntaxhighlight>


Note that {{inlinecode|groups(0)}} means a completely different kind of thing
: There ''is'' an argument there, but it's not an index, it's the default value for optional captures (the default default is None)
: for example, to hijack an example from the docs:
<syntaxhighlight lang="python">
m = re.match(r"(\d+)\.?(\d+)?", "24")
m.groups()    == ('24', None)
m.groups('0') == ('24', '0')    # so 0 instead of '0' means putting an ''integer'' there, which is often not what you want
</syntaxhighlight>


...and more.

<!--
* Match.groupdict()
: can help you catch things into names specified in the regexp
-->

===Examples===


'''If you just want to know the piece of text that was matched overall (regardless of capturing)'''
<syntaxhighlight lang="python">
>>> s = '   a 11   '
>>> m = re.search(r'\b(a|b|c) [0-9]+\b', s)
>>> m.groups()
('a',)
>>> m.group(1)    # the number wasn't captured, but is still patch of the match
'a'
>>> m.group(0)    # that number was still part of the overall match (0 is a special cse)
'a 11'
>>> st,en = m.span() # alternatively...   (e.g. handy if you want to show the match in context again)
>>> st, en
(3, 7)
>>> s[st:en]
'a 11'
</syntaxhighlight>



'''If you want all matches in a string'''

If you run the above on 
<syntaxhighlight lang="python">
s = '   a 11   c 22 '
</syntaxhighlight>
you will find that search() will only match just one. 

findall() will give you the captured strings
<syntaxhighlight lang="python">
re.findall(r'\b(a|b|c) [0-9]+\b', s) == ['a', 'c']
</syntaxhighlight>

...which may be close to what you want (with a few changes of the capturing), but if not, you probably want a match object for each, e.g.
<syntaxhighlight lang="python">
list( re.finditer(r'\b(a|b|c) [0-9]+\b', s) ) 
        == [ <re.Match object; span=(3, 7), match='a 11'>,    <re.Match object; span=(10, 14), match='c 22'> ]
</syntaxhighlight>




'''Other stuff'''

You can ''match'' wildcard repetition, but you cannot ''extract'' each individual wildcard match.  For example, 
m = re.match('([0-9][0-9])+', '12345678')  # with the idea of matching two numbers at a time
<syntaxhighlight lang="python">
m.groups() == ('78',)
m.group(0) == '12345678'
m.group(1) == '78'
</syntaxhighlight>

Workarounds
* repeatedly match (though for various cases, regexp add very little)

* consider re.findall() 
: not equivalent at all - it will skip over character is doesn't match

===Split details===
re.'''split'''() can include or exclude what you split on, basically depending on whether capturing parentheses are used:
<syntaxhighlight lang="python">
re.split('(-)', '1-800-1234') == ['1', '-', '800', '-', '1234']
re.split('-',   '1-800-1234') == ['1', '800', '1234']

# Note that because they'll always be every second string, it's easy capture and selectively ignore separators, e.g.:
m = re.split('([ -])',   '1-800 1234')   
m       == ['1', '-', '800', ' ', '1234']
m[::2]  == ['1', '800', '1234']
m[1::2] == ['-', ' ']
</syntaxhighlight>


===Lookahead and lookbehind===

Lookahead and lookbehind are useful to check something without consuming it.

Note that, dending on case, they can mean less efficient processing, so avoid them when you easily can.


'''Positive lookbehind''' (<tt>?&lt;=</tt>), '''positive lookahead''' (<tt>?=</tt>):
<syntaxhighlight lang="python">
#remove dashes only if inside a number (that is: digit on both sides)
re.sub('(?<=[0-9])[\-](?=[0-9])', '', '-1-800-1234-') == '-18001234-'

#Somewhat trickier use: force a space after each letter-dot pair
re.sub(r'(?<=[A-Z])\.(?!\s)', '. ', 'A.N.Smith.') == 'A. N. Smith.'
</syntaxhighlight>

<syntaxhighlight lang="python">
# Match 'Asimov' only if preceded by 'Isaac' (numbers to see which one it matches)
re.findall('((?<=Isaac )Asimov[0-9])', 'Asimov1 Isaac Asimov2 Asimov3')
# gives:
['Asimov2']
</syntaxhighlight>
Note: Lookbehind requires fixed-length expressions, so <tt>(?<=Isaac\s+)Asimov</tt> is invalid.


<syntaxhighlight lang="python">
#Try to find entity names in raw, possibly badly formed HTML,
# but avoid things where the & seems to be part of a URL query string
re.findall('&([\#A-Za-z0-9]+)(?=[;\s\.\,])', '&amp; &bad. &#234 link?a=1&b=2&coo=3') 
# gives:
['amp', 'bad', '#234']
</syntaxhighlight>


'''Negative lookbehind''' (<tt>?&lt;!</tt>), '''negative lookahead''' (<tt>?!</tt>): 
<syntaxhighlight lang="python">
#remove dashes not at all adjacent to numbers:
re.sub('(?<![0-9])[\-](?![0-9])', '', '- -1-800-1234- - -1') 
# gives:
' -1-800-1234-  -1'
</syntaxhighlight>


'''Non-grouping parentheses''' (<tt>?:</tt>) are useful when you want to avoid nested results, or avoid capturing certain chunks as results at all (but still use ?, *, +, {} or such on it): 
<syntaxhighlight lang="python">
re.findall('&amp;((#x|x|#)?[A-Za-z0-9]+;)', '&amp;x33; &amp;#33; &amp;#x33; &33;')
# gives:
[('&amp;x33;', 'x'), ('&amp;#33;', '#'), ('&amp;#x33;', '#x'), ('&amp;#33;', '')]

#while:

re.findall('(&amp;(?:#x|x|#)?[A-Za-z0-9]+;)', '&amp;x33; &amp;#33; &amp;#x33; &33;')
# gives:
['&amp;x33;', '&amp;#33;', '&amp;#x33;', '&33;']
</syntaxhighlight>


===Raw strings===


Because 
* backslash is used for escape sequences in strings (in python and many other languages)
* combinations of a backslash character and another character has special meaning to regular expressions
...you will run into some cases of confusion.

While this is not ''purely'' for regexps (it's also useful for paths on windows), it's one of the main uses. 
: ...to the point that some editors will treat a raw strings as "syntax highlight this as regexps"



For example, 
 '\b' 
is the bell character (0x0d), while if you wanted to test for word border, you probably meant the two characters backslash and r. You can do:
 '\\r'
When using raw strings, you can also write that as:
 r'\r'


There are only some backslash escapes that are specially interpreted, so everything else will be seen as the two separate characters. 
As such, you can get away with it in either form for cases like:
* '\s' (=='\\s')
* brackets

There are things that are not safe, but are not not used in regexps so you probably won't make them into problems:
* {{inlinecode|\t}}
* {{inlinecode|\f}}
* {{inlinecode|\n}}, except most people ''expect'' that to be newline so it's not generally an issue

The main cases you should worry about are probably
* {{inlinecode|\"}} and {{inlinecode|\'}}  (also re)
* {{inlinecode|\r}}
* {{inlinecode|\b}}


[[Category:Python]]