Python notes - syntax and language - changes and py2/3
Syntaxish: syntax and language · changes and py2/3 · decorators · importing, modules, packages · iterable stuff · concurrency
IO: networking and web · filesystem Data: Numpy, scipy · pandas, dask · struct, buffer, array, bytes, memoryview · Python database notes Image, Visualization: PIL · Matplotlib, pylab · seaborn · bokeh · plotly
Stringy: strings, unicode, encodings · regexp · command line argument parsing · XML speed, memory, debugging, profiling · Python extensions · semi-sorted |
Contents
On preparing for and doing the conversion to py3
It is usually easy (but sometimes awkward) to write code that is good for both ~py2.7 and py3k, which was nice preparation for the shift.
2to3 can do most of the syntax for you, but it cannot really think about how you mixed bytestrings and unicode strings - such are the drawback of weak typing when you change your entire language's semantics.
Also, various third party libraries decided to move from bytes by default to unicode by default -- which is effectively an API change it cannot know about.
So you will have to review your code.
At one point I changed all my code to run under py3, to force myself to do all conversions over a weekend.
This was work, but the breaking changes were actually mostly boring, most changes being:
- sorting, where I wasn't using key= yet
- the equivalent key= form is typically obvious
- bytes/str around the edges of the app, e.g.
- file IO
- web framework
- database
- network library renames
Differences in syntax and behaviour in 2.6, 3k
This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, or tell me) |
NOTE: These are notes to self, basically "things that bit me that I looked up"
Syntax differences
- py2.6: asbecame a reserved word
- py2.6: withbecame a reserved word
- py3: print() is a function
- so print("stuff") instead of print "stuff"
- and some tricks (like the comma to omit a newline) changed
- more details below
- py3: no more backticks, use repr() explicitly
- py3: int and long are now (syntactically) the same thing, so you can use int everywhere
- py3: Division
- In py2
- /
- coerces to float if it involves a float, e.g. 1/2.0==0.5 (it's not unusual to put float() around variable arguments to ensure you get float division)
- stays an int if both arguments were int, e.g. 1/2==0
- //
- integer division, basically. E.g. 1//2==0
- result type is still float if involves a float but the behaviour is no different, e.g. 1//2.0==0.0
-
- In py3
- /
- always float division, e.g. 1/2==0.5
- //
- as in py2
-
- In py2
- Exceptions now always use as , e.g. except ErrorName as var:
- as was possible for a bunch of versions, but the older except ErrorName, var:was also still allowed
- for more details: http://www.python.org/dev/peps/pep-3110/
- py3: no more has_key
- The inkeyword should handle all cases.
- py3: no more g.next()
- It was renamed to g.__next__()because it's considered an internal method.
- your code should probably say next(g)rather than either of those.
- py3: no more <>, use != instead
- py3: you can no longer mix tabs and spaces. Probably a good thing.
- py3.6: allows type annotation (see below)
Behaviour/implementation changes
- py3: list comprehensions no longer leak their iteration variables
- because they are actually functions with their own scope now
- py3: more things return an iterator or view, rather than a list (which is nice)
- dict.keys(), dict.items() and dict.values() (and dict.iterkeys(), dict.iteritems() and dict.itervalues() are gone)
- map() and filter() return iterables
- range() acts like the older xrange (and xrange is gone)
- zip returns an iterator
- py2 had .next() for iterators, py3 has a next() in syntax
- py3: comparisons between types is slightly stricter
- less likely to do strange things without warning, and some related details (no maxint, no L in int's repr
- TODO: detail
- py3: bytestrings / unicode strings are their own topic, see below
- buffer, memoryview and such changed.
- largely compatible, but some details have changed
- if you use them, read up to be sure
- py3: has new-style classes only (that is, inherit from object) (no more classic classes)
- which most people were using already
See also:
- 2to3 is a tool to parse code and show suggested change (as a diff)
- http://docs.python.org/3.0/whatsnew/3.0.html
- http://wiki.python.org/moin/Python3000
- http://wiki.python.org/moin/FutureProofPython
- http://oakwinter.com/code/porting-setuptools-to-py3k/
Networking changes
Various things have been reorganized, but are largely unchanged::
- httplib has been moved to http.client
- urllib2 has been moved to mostly into urllib.request and urllib.error
- as before, consider things like requests - not standard library, but convenient.
- urlparse became urllib.parse
- urlencode, quote, quote_plus, unquote moved from urllib to urllib.parse
- cgi.escape became html.escape
TODO: I think timeout details were cleaned up(verify)
See also Python_usage_notes/Networking_and_web#URL_fetching
Py2 versus Py3 string stuff
If you're new to python, maybe just learn the py3 way, and don't read this - there's no point in confusing yourself with both parts of a conflicting change.
Types and literals
- The distinctions in py2 and py3 are the same
- there's a bytestring
- there's a unicode string
- What has changed is their names, and which is the default
- strin py2 meant bytestring,strin py3 means unicode
- put another way, in py2:
- stris bytestring type, a literal looks like''
- unicodeis unicode string type, a literal looks likeu''
-
- and in py3:
- bytesis bytestring type, a literal looks likeb''
- stris unicode string type, a literal looks like''
-
Changes in code - conversions
- unicode from integer codepoints
- py2: unichr(i)
- because chr() returned a bytestring so could only deal with 0..255
- py3: chr(i)
- bytestring from integer(s)
- py2: chr(i)
- py3: bytes([i])
- If you want "bytes from integers" with syntax that that works in both py2 and py3
- you will find that bytes([i]) is valid code in py2 but does something else
- there's a duct tape fix in bytes(bytearray([i])), though note it won't be the most efficient (bytearray, since ge;py2.6, is a mutable counterpart to bytes)
text conversions (bytestring-unicode)'
- bytes to unicode:
- py2: ''.decode(encoding)
- py3: b''.decode(encoding)
- py2:
- unicode to bytes:
- py2: u''.encode(encoding)
- py3: ''.encode(encoding)
- py2:
Note that
- converions during initialization are also a thing, e.g.
- initialize a bytes object from unicode and an encoding (seems equivalent to str.encode(verify)), e.g. bytes('something', 'utf-8') == b'something'
- initialize a unicode object from bytes and an encoding: (seems equivalent to bytes.decode(verify)), e.g. str(b'something', 'utf-8') == 'something'
- py2 let you .encode() a bytestring (seemed intended for codepage stuff?), this was a somewhat confusing exception,
and py3 does not allow you to .encode a bytes object.
- py2 also allowed some byte-to-byte codings, that are now only available via the codecs module:
byte conversions
codecs's codecs.encode() and codecs.decode(), aside from text encodings, also includes bytes-to-bytes things (found only here), including:
- 'hex' / 'hex_codec'
- 'base64' (Base64)
- 'bz2'
- 'zlib'
- 'uu' / 'uu_codec' (UUEncoding)
- 'quopri' / 'quopri_codec' (quoted printable)
Iterating
py2:
list('1234') == ['1', '2', '3', '4'] list(u'1234') == [u'1', u'2', u'3', u'4']
py3:
list(b'1234') == [49, 50, 51, 52] list('1234') == ['1', '2', '3', '4']
Coercion
py2:
- allowed coercing bytestrings to unicode e.g. u'\u12d6' + b'foo' == u'\u13d6foo'
py3:
- the same raises a TypeError
- forces you to be explicit, e.g. doing an encode or decode to work on the same type
While that leads to a little more typing in py3, it's also less fuzzy on responsibilities and less sensitive to typing errors
It also makes it harder to write functions that work on both bytestrings and unicode strings, but that's arguably sort of the point.
Brackets
- python2 allowed print without brackets
- ...which was an almost-singular exception in terms of syntax
- python3 makes it print(), like all other functions. It's a syntax error not to
- it also has some keyword arguments:
- end: string appended after the last value, default a newline (allows imitating the py2-comma-at-the-end)
- sep: string inserted between values (default a space)
- file: output to a file-like object (defaults to sys.stdout)
- flush: whether to forcibly flush the stream (default false)
print without newline
py2:
print 'foo',
py3:
print('foo', end='')
If you want something that works in both py2 and py2, use sys.stdout.write()
Sequences and non-string objects
in both py2 and py3, non-string objects will basically be asked for their .__str__()
Note that
- py2 with bracket would basically be interpreted as a tuple argument
print(1,2,3,4)
- py3 would interpret the same as multiple (non-keyword) arguments -- but typically amount to the same output
print(1,2,3,4)
in py2, the separator was a single space(verify), in py3, it can be specified:
print(1,2,3,4, sep='\t')
in py2, output was to sys.stdout, in py3 you can specify any file-like object
print("WARNING" file=sys.stderr)
- e.g. for stderr means you don't have to use its write() instead of print
py2, explicit flushing meant calling sys.stdout.flush(), in py3 you can do:
print(' ', flush=True)
Unsorted:
- ≥py2.6 can do from __future__ import unicode_literals(verify)
- meaning what exactly?
- python2 added bytesas an alias for str around 2.7
- meaning what exactly?
- some easing of transition:
- ≥py2.something accepts bandbytes, aliased to its str type (bytestring)
- ≥py3.3 accepts uas an alias for its str (unicode)
- ≥py2.something accepts
Keep in mind that print is geared towards printing (unicode) text,
and puts everything through io.TextIOBase
If you want to output raw bytes, you'll basically need sys.stdout.write instead.
Console and locale details
This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, or tell me) |
and it comes with more wrapping code than most other IO.
It will, for example, convert to the locale -- which could fail if that's a codepage or 'ascii'
sys.stdout.encoding is what is set for that specific stream
- CHECK: on windows this is now UTF-8 (previously was codepages, behaviour you can still get with PYTHONLEGACYWINDOWSSTDIO ? TODO: read https://www.python.org/dev/peps/pep-0528/
- on unices this is based on your locale ([1][2]), which makes sense for console
- on a lot of locales it amounts to UTF-8 so will work well and combine well -- but more because of convention, and you may not want to count on this
- if it ends up being 'ascii', it's probably the main reason you've seen "'ascii' codec can't encode characters in position ...: ordinal not in range"
- ...so there are use cases where you want to force it to something (probably UTF-8), like piping programs together.
- [3] overrides stdin/stdout/stderr
- you can't set it from code - setting it throws an exception http://www.macfreek.nl/memory/Encoding_of_Python_stdout#Overriding_the_Encoding_of_stdout_or_stderr
- locale.getpreferredencoding() (for unix-style locales)
- ...as an indication of what the console uses, but it's basically used for that already
sys.getdefaultencoding()
- used for implicit encodings (so more in py2), as a system-wide default and nothing better implied by context
- but was 'ascii' on py2 unless overridden -- and there are good (breaky) reasons you never want to do that [4]
- seems to often be utf-8 in py3?
sys.getfilesystemencoding()
- intended for filenames, command-line arguments, environment variables
- often utf-8?
https://pythonhosted.org/kitchen/unicode-frustrations.html
-->
Paths
Consider pathlib
IO
This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, or tell me) |
network IO and file IO is bytes by nature so often involves either a lot of explicit conversions (or "I'll do everything for you" libraries).
file open()
py2 had file() and open() (The latter is the same as io.open) which did mostly the same thing, but with a somewhat different interface.
py3 has only open()
open() also controls what type read() will give you.
py2:
- open() gives bytes
py3:
- open() in text mode (without 'b' in the mode): gives unicode
- has an encoding parameter, if not specified will default to locale.getpreferredencoding(False), likely but not given to be 'UTF-8'
- open() in binary mode (with 'b' in the mode): gives bytes (and specifying encoding will raise ValueError)
subprocess:
py2:
- pipes are bytes
subprocess.Popen('ls', stdout=subprocess.PIPE,stderr=subprocess.PIPE).communicate() ('\xe1\x8f\x96\n', )
py3:
- pipes are bytes unless you explicitly specify an encoding
subprocess.Popen('ls', stdout=subprocess.PIPE,stderr=subprocess.PIPE).communicate() (b'\xe1\x8f\x96\n', b) subprocess.Popen('ls', stdout=subprocess.PIPE,stderr=subprocess.PIPE, encoding='utf8').communicate() ('Ꮦ\n', )
Bytes and strings around file IO
This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, or tell me) |
- py3: no more file()
- and the built-in 'open() (the synonym file() no longer exists) grew some extra behaviour:
- to read/write strings, automatically converted from their encoded form, specify an encoding. The default encoding is platform-dependent, so you probably don't want to rely on that.
- to read/write bytes, specify a binary mode, ('rb', 'wb') and don't specify an encoding
On in-memory versions:
- StringIO acts like a file opened in text mode,
- BytesIO acts like a file opened in binary mode.
See also:
misc
b'aa'.decode('hex_codec')
will tell you
LookupError: 'hex_codec' is not a text encoding; use codecs.decode() to handle arbitrary codecs
Instead, probably do (works on unicode and bytes input):
import codecs codecs.decode(b'aa', 'hex_codec')
Type annotation
See Python_notes_-_syntax_and_language#Type_annotation
StringIO, and now BytesIO
For context / basic use
Python2 had StringIO objects, objects that acted like file objects you could write() bytestrings into.
Is is mostly useful where a function wants to write to file object, but you want to avoid the filesystem, e.g. for speed by avoiding IO, or to avoid potential permission problems.
Example:
sio = StringIO.StringIO() Image.save(sio) data = sio.getvalue()
Notes:
- you can write(), but you cannot read() -- you can only getvalue() the full contents so far
- Once the StringIO object is close()d, the contents are gone
- not usually a problem, as most save functions either take a filename and does an open()-write()-close() (in which case stringio is fairly irrelevant), or take a file object and just write() (in which case you're fine)
- (could apparently also take unicode, but most people use it for bytes?)
StringIO was a python implementation in its standard library, cStringIO was a faster C implementation and drop-in, so you'ld often see
try: # use the faster extension when we can import cStringIO as StringIO except: # drop back to python's own when we must import StringIO
Python3 changes
Since bytestrings are now called bytes and not str, the same functionality is now in- which behaves the same way
- though BytesIO also lets you have a read-write view[5]
- which py2 did not[6]
Python3 also has has a similar thing for unicode strings.
- io.TextIOWrapper is a seekable, file-like object, that also implies some conversions
- io.StringIO, which is basically a binary unicode representation
https://docs.python.org/3/library/io.html