Python notes - syntax and language - changes and py2/3

From Helpful
Jump to: navigation, search
Syntaxish: syntax and language · changes and py2/3 · importing, modules, packages · iterable stuff · concurrency

IO: networking and web · filesystem

Data: Numpy, scipy · pandas, dask · struct, buffer, array, bytes, memoryview · Python database notes

Image, Visualization: PIL · Matplotlib, pylab · seaborn · bokeh · plotly


Concurrency (threads, processes, more) · joblib · pty and pexpect

Stringy: strings, unicode, encodings · regexp · command line argument parsing · XML

date and time

speed, memory, debugging, profiling

semi-sorted



On preparing for and doing the conversion to py3

It is usually easy (but sometimes awkward) to write code that is good for both ~py2.7 and py3k, which was nice preparation for the shift.


2to3 can do most of the syntax for you, but it cannot really think about how you mixed bytestrings and unicode strings - such are the drawback of weak typing when you change your entire language's semantics.

Also, various third party libraries decided to move from bytes by default to unicode by default -- which is effectively an API change it cannot know about.

So you will have to review your code.


At one point I changed all my code to run under py3, to force myself to do all conversions over a weekend.

This was work, but the breaking changes were actually mostly boring, most changes being:

  • print
  • sorting, where I wasn't using key= yet
the equivalent key= form is typically obvious
  • bytes/str around the edges of the app, e.g.
    • file IO
    • web framework
    • database
  • network library renames


Differences in syntax in 2.6, 3k

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

NOTE: These are notes to self, basically "things that bit me that I looked up"


Syntax differences

  • py2.6:
    as
    became a reserved word
  • py2.6:
    with
    became a reserved word
  • py3: print() is a function
so print("stuff") instead of print "stuff"
and some tricks (like the comma to omit a newline) changed
more details below
  • py3: no more backticks, use repr() explicitly
  • py3: int and long are now (syntactically) the same thing, so you can use int everywhere
  • py3: Division
In py2
/
coerces to float if it involves a float, e.g. 1/2.0==0.5 (it's not unusual to put float() around variable arguments to ensure you get float division)
stays an int if both arguments were int, e.g. 1/2==0
//
integer division, basically. E.g. 1//2==0
result type is still float if involves a float but the behaviour is no different, e.g. 1//2.0==0.0
In py3
/
always float division, e.g. 1/2==0.5
//
as in py2
  • Exceptions now always use as , e.g.
    except ErrorName as var:
as was possible for a bunch of versions, but the older
except ErrorName, var:
was also still allowed
for more details: http://www.python.org/dev/peps/pep-3110/
  • py3: no more has_key
The
in
keyword should handle all cases.
  • py3: no more
    g.next()
It was renamed to
g.__next__()
because it's considered an internal method.
your code should probably say
next(g)
rather than either of those.


  • py3: no more <>, use != instead
  • py3: you can no longer mix tabs and spaces. Probably a good thing.
  • py3.6: allows type annotation (see below)



Behaviour/implementation changes

  • py3: list comprehensions no longer leak their iteration variables
because they are actually functions with their own scope now
  • py3: more things return an iterator or view, rather than a list (which is nice)
    • dict.keys(), dict.items() and dict.values() (and dict.iterkeys(), dict.iteritems() and dict.itervalues() are gone)
    • map() and filter() return iterables
    • range() acts like the older xrange (and xrange is gone)
    • zip returns an iterator
py2 had .next() for iterators, py3 has a next() in syntax
  • py3: comparisons between types is slightly stricter
less likely to do strange things without warning, and some related details (no maxint, no L in int's repr
TODO: detail
  • py3: bytestrings / unicode strings are their own topic, see below
  • buffer, memoryview and such changed.
largely compatible, but some details have changed
if you use them, read up to be sure
  • py3: has new-style classes only (that is, inherit from object) (no more classic classes)
which most people were using already


See also:

  • 2to3 is a tool to parse code and show suggested change (as a diff)


Networking changes

Various things have been reorganized, but are largely unchanged::

  • httplib has been moved to http.client
  • urllib2 has been moved to mostly into urllib.request and urllib.error
as before, consider things like requests - not standard library, but convenient.
  • urlparse became urllib.parse
  • urlencode, quote_plus moved from urllib to urllib.parse
  • cgi.escape became html.escape


TODO: I think timeout details were cleaned up(verify)


See also Python_usage_notes/Networking_and_web#URL_fetching



Py2 versus Py3 string stuff

If you're new to python, maybe just learn the py3 way, and don't read this - there's no point in confusing yourself with both parts of a conflicting change.


From a distance, the distinctions in py2 and py3 are the same:

there's a bytestring
there's a unicode string

What's changed

their names
the default.
str
used to be bytestring, now it's unicode string
string literals without qualifiers used to be bytestring, and are now unicode

There's a little more to it than that, though.


unicode versus bytes types

py2:

  • str
    is bytestring type, a literal looks like
    ''
  • unicode
    is unicode string type, a literal looks like
    u''

py3:

  • bytes
    is bytestring type, a literal looks like
    b''
  • str
    is unicode string type, a literal looks like
    ''



unicode from integer codepoints

  • py2:
    unichr(i)
because chr() returned a bytestring so could only deal with 0..255
  • py3:
    chr(i)


bytestring from integers

  • py2:
    chr(i)
  • py3:
    bytes([i])

If you want "bytes from integers" with syntax that that works in both py2 and py3

you will find that bytes([i]) is valid code but does something else
there's a duct tape fix in
bytes(bytearray([i]))
, though note it won't be the most efficient (bytearray, since ge;py2.6, is a mutable counterpart to bytes)


bytestring-unicode conversions'

bytes to unicode:

  • py2:
    u''.encode(encoding)
  • py3:
    ''.encode(encoding)

unicode to bytes:

  • py2:
    u''.encode(encoding)
  • py3:
    ''.encode(encoding)


In python3, you can also

  • initialize a bytes object from unicode and an encoding (seems equivalent to str.encode)
note that this requires an encoding
bytes('something', 'utf-8') == b'something'
  • initialize a unicode object from bytes and an encoding: (seems equivalent to bytes.decode)
str(b'something', 'utf-8') == 'something'

This isn't quite equivalent(verify) to

'something'.encode('u8')
b'something'.decode('u8')

...but it's close enough in a lot of practice, and you may find the latter more intuitive.


Iterating

py2:

list('1234')  == ['1', '2', '3', '4']
list(u'1234') == [u'1', u'2', u'3', u'4']

py3:

list(b'1234') == [49, 50, 51, 52]
list('1234') == ['1', '2', '3', '4']


Coercion

py2:

allowed it,
u'\u12d6' + b'foo' == u'\u13d6foo'

py3

the same raises a TypeError
forces you to be explicit, e.g. doing an encode or decode to work on the same type

While that leads to a little more typing in py3, it's also clearer responsibilities


(...and making it harder to write functions that work on both bytestrings and unicode strings, but that's sort of the point. The only case where I can think of you wanting both is low level HTTP stuff, but even that is arguable)


encode() a bytestring?

py2: you can .encode() a py2 str, which seems intended for codepage stuff?

a somewhat confusing exception to the typing.


py3: you can't .encode a py3 bytes



print

Brackets

  • python2 allowed print without brackets
...which was an odd exception in terms of syntax
  • python3 makes it print(), like all other functions. It's a syntax error not to
it also has some keyword arguments:
end: string appended after the last value, default a newline (allows imitating the py2-comma-at-the-end)
sep: string inserted between values (default a space)
file: output to a file-like object (defaults to sys.stdout)
flush: whether to forcibly flush the stream (default false)


print without newline

py2:

print 'foo',

py3:

print('foo', end='')

If you want something that works in both py2 and py2, use sys.stdout.write()


Sequences and non-string objects

in both py2 and py3, non-string objects will basically be asked for their .__str__()


Note that

py2 with bracket would basically be interpreted as a tuple argument
print(1,2,3,4)
py3 would interpret the same as multiple (non-keyword) arguments -- but typically amount to the same output
print(1,2,3,4)



in py2, the separator was a single space(verify), in py3, it can be specified:

print(1,2,3,4, sep='\t')

in py2, output was to sys.stdout, in py3 you can specify any file-like object

print("WARNING" file=sys.stderr)
e.g. for stderr means you don't have to use its write() instead of print

py2, explicit flushing meant calling sys.stdout.flush(), in py3 you can do:

print(' ', flush=True)


You can get py3 style print() in py2, since py2.6(verify) via
from __future__ import print_function



Unsorted:

  • ≥py2.6 can do
    from __future__ import unicode_literals
    (verify)
meaning what exactly?
  • python2 added
    bytes
    as an alias for str around 2.7
meaning what exactly?
  • some easing of transition:
    • ≥py2.something accepts
      b
      and
      bytes
      , aliased to its str type (bytestring)
    • ≥py3.3 accepts
      u
      as an alias for its str (unicode)




Keep in mind that print is geared towards printing (unicode) text, and puts everything through io.TextIOBase

If you want to output raw bytes, you'll basically need sys.stdout.write instead.


Console and locale details
This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

Remember, print is IO that is specific to the console, so its behaviour depends on the environment.

It will, for example, convert to the locale -- which can fail if that's a codepage or 'ascii'


sys.stdout.encoding is what is set for that specific stream

CHECK: on windows this is now UTF-8 (previously was codepages, behaviour you can still get with PYTHONLEGACYWINDOWSSTDIO ? TODO: read https://www.python.org/dev/peps/pep-0528/
on unices this is based on your locale ([1][2]), which makes sense for console
on a lot of locales it amounts to UTF-8 so will work well and combine well -- but more because of convention, and you may not want to count on this
if it ends up being 'ascii', it's probably the main reason you've seen "'ascii' codec can't encode characters in position ...: ordinal not in range"
...so there are use cases where you want to force it to something (probably UTF-8), like piping programs together.
[3] overrides stdin/stdout/stderr
you can't set it from code - setting it throws an exception http://www.macfreek.nl/memory/Encoding_of_Python_stdout#Overriding_the_Encoding_of_stdout_or_stderr


locale.getpreferredencoding() (for unix-style locales)
...as an indication of what the console uses, but it's basically used for that already


sys.getdefaultencoding()

used for implicit encodings (so more in py2), as a system-wide default and nothing better implied by context
but was 'ascii' on py2 unless overridden -- and there are good (breaky) reasons you never want to do that [4]
seems to often be utf-8 in py3?



sys.getfilesystemencoding()

intended for filenames, command-line arguments, environment variables
often utf-8?


PYTHONLEGACYWINDOWSFSENCODING


https://pythonhosted.org/kitchen/unicode-frustrations.html


-->

Paths

IO

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

network IO and file IO is bytes by nature so often involves either a lot of explicit conversions (or "I'll do everything for you" libraries).


file open()

py2 had file() and open() (The latter is the same as io.open) which did mostly the same thing. py3 has only open()


open also controls what type read() will give you.

py2:

open() gives bytes

py3:

open() in text mode (without 'b' in the mode): gives unicode, encoding defaults to locale.getpreferredencoding(False), likely to be 'UTF-8'
open() in binary mode (with 'b' in the mode): gives bytes (also specifying encoding will raise ValueError)



subprocess:

py2:

pipes are bytes
subprocess.Popen('ls', stdout=subprocess.PIPE,stderr=subprocess.PIPE).communicate()
('\xe1\x8f\x96\n', )


py3:

pipes are bytes unless you explicitly specify an encoding
subprocess.Popen('ls', stdout=subprocess.PIPE,stderr=subprocess.PIPE).communicate()
(b'\xe1\x8f\x96\n', b)
subprocess.Popen('ls', stdout=subprocess.PIPE,stderr=subprocess.PIPE, encoding='utf8').communicate()
('Ꮦ\n', )




Bytes and strings around file IO

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)


  • py3: no more file()
  • and the built-in 'open() (the synonym file() no longer exists) grew some extra behaviour:
  • to read/write strings, automatically converted from their encoded form, specify an encoding. The default encoding is platform-dependent, so you probably don't want to rely on that.
  • to read/write bytes, specify a binary mode, ('rb', 'wb') and don't specify an encoding


On in-memory versions:

  • StringIO acts like a file opened in text mode,
  • BytesIO acts like a file opened in binary mode.


See also:

misc

b'aa'.decode('hex_codec')

will tell you

LookupError: 'hex_codec' is not a text encoding; use codecs.decode() to handle arbitrary codecs

Instead, probably do (works on unicode and bytes input):

import codecs
codecs.decode(b'aa', 'hex_codec')

StringIO, and now BytesIO

For context / basic use

Python2 had StringIO objects, objects that acted like file objects you could write() bytestrings into.

Is is mostly useful where a function wants to write to file object, but you want to avoid the filesystem, e.g. for speed by avoiding IO, or to avoid potential permission problems.

Example:

sio = StringIO.StringIO()
Image.save(sio)
data = sio.getvalue()


Notes:

  • you can write(), but you cannot read() -- you can only getvalue() the full contents so far
  • Once the StringIO object is close()d, the contents are gone
not usually a problem, as most save functions either take a filename and does an open()-write()-close() (in which case stringio is fairly irrelevant), or take a file object and just write() (in which case you're fine)
  • (could apparently also take unicode, but most people use it for bytes?)


StringIO was a python implementation in its standard library, cStringIO was a faster C implementation and drop-in, so you'ld often see

try: # use the faster extension when we can
    import cStringIO as StringIO
except: # drop back to python's own when we must
    import StringIO



Python3 changes

Since bytestrings are now called bytes and not str, the same functionality is now in
io.BytesIO
which behaves the same way
though BytesIO also lets you have a read-write view[5]
which py2 did not[6]


Python3 also has has a similar thing for unicode strings.

io.TextIOWrapper is a seekable, file-like object, that also implies some conversions
io.StringIO, which is basically a binary unicode representation


https://docs.python.org/3/library/io.html


Type annotation

Python 3.6 introduced a syntax for annotating variables as described in PEP 526 (though a bunch more PEPs are relevant)

It looks like

def greeting(name: str) -> str:
    return 'Hello ' + name


You can also type variables, like

i:int = 1


In practice, this is type annotation that isn't used for anything at runtime.

It specifically isn't type checking. It's basically a comment, but one more parseable by IDEs, so that they can show it to programmers.

Which is useful, but you should assume it might even be lying, and inspect the code.


And yes, MyPy can enforce it, but this is probably only used in tests. Tests that probably make your code better, but only indirectly. If you want the actual safety of a statically typed language, use a statically typed language.