The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Syntaxish: syntax and language · changes and py2/3 · decorators · importing, modules, packages · iterable stuff · concurrency

IO: networking and web · filesystem

Data: Numpy, scipy · pandas, dask · struct, buffer, array, bytes, memoryview · Python database notes

Image, Visualization: PIL · Matplotlib, pylab · seaborn · bokeh · plotly

Tasky: Concurrency (threads, processes, more) · joblib · pty and pexpect

Stringy: strings, unicode, encodings · regexp · command line argument parsing · XML

date and time

Notebooks

speed, memory, debugging, profiling · Python extensions · semi-sorted

Exceptions

args, str, and such

Exceptions have an args variable, which is a tuple.

Often with a single string, sometimes more data-like details.

is set via BaseException.__init__(*args)

read out by BaseException.__str__

use/meaning is mostly up to you.

It makes it easy to put a message and/or arbitrary values in there with minimal work:

class MyException(Exception):
    pass

try:
    raise MyException("bad thing",1.9)
except Exception, e:
    print e.args
    print repr(e)
    print str(e)

Specific exceptions may add some more structured details elsewhere. For example, OSError adds errno.

Be careful when assuming what's in args. It's probably good style to only look for values when catching very specific exceptions - and note that in some cases, names/details have changed over time.

Printing an exception:

repr() is most informative, though sometimes a little clunky.
str() mostly just prints args (and taking out some brackets when len(args)==1)

many people put a readable thing in args[0]

e.message (on BaseException) was introduced and then deprecated

don't assume it's there, don't use it

https://docs.python.org/2/library/exceptions.html

https://wiki.python.org/moin/HandlingExceptions

https://julien.danjou.info/blog/2016/python-exceptions-guide -->

Catching more than one exception

As you probably know, catching distinct exceptions is not a special case:

except socket.error, e:
    print 'Socket error: %s'%str(e)
except httplib.HTTPException:
    print 'HTTP problem.'

In some situations, you want to react the same way, e.g. catching multiple network-related exceptions and just say "network was stupid" message.

What you are looking for is:

except (socket.error, urllib2.URLError, httplib.HTTPException), e:
    # when you want to log the error, you may want to 
    # mention the exception type as well as its arguments, so:
    print 'Networking problem, %s: %s'%(e.__class__, str(e))

And not:

 except Exception1, Exception2:

This is actually just the earlier form, and means "if Exception1 is caught, bind Exception2 as a variable name in local scope that refers to the exception object" - also meaning Exception2 is never caught by this.

Since python 2.6 you can use as instead (and py3 accepts only as), which avoids the last ambiguity, and can read a little easier:

except (Exception1, Exception2) as e:

try, except, and finally

The try and except, as a pattern, is generally known.

finally is also interesting: you can specify a code block that is always executed, including when the containing code returns, excepts, or uses continue or break.

Finally is a useful way to do cleanup that is not conditional, both in terms of execution and code cleanliness. You only have to put it in your code once, not on each separate path out of a block (or, regularly, function).

In ≤py2.4 you had to end a try block with either except or finally, so if you wanted to use finally for cleanup, for example for database transactions and/or connections, you might have written:

try:
    dbconn = connect()
    try:
        #your code, probably followed by a
        dbconn.commit()
    except SomeError:
        dbconn.rollback()
finally: #cleanup, such as
    dbconn.close()

From python 2.5 on, you can write:

try:
    #your code, probably followed by a
    dbconn.commit()
except SomeError:
    dbconn.rollback()
finally:
    dbconn.close()

...and else

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

try:
       # code
except:
       # executed if code raised an exception
else:
       # executed if try block finishes without exception
finally:
       # executed regardless of exception

The code you put in the else block is executed only if execution flows out of the try block naturally, so it will not execute if there was an exception, return, continue, or break.

...so it differs from finally in that

Finally still executes if there was an return, continue, or break, and regardless of whether there was an exception or not

else executes only when none of those happened.

But perhaps more importantly, it differs from the main code block in that

exceptions in the else block is not caught by the except.

What is this good for?

I have seen a lot of vague and/or poor examples, where it's just different rather than clearly better.

Most of the decent examples I've seen involve exception handling - better isolating what you're handling. And yes, those examples can be written equivalently with just try-excepts-finally.

Some examples also make a decent case that the blocks and flow might stay a little more compact and/or readable.

"How does that differ from putting the code inside the try?"

Depending on what the code is, it may belong there, yes.

The exception-y reason is that the try block can trip into the except blocks, the else block cannot, and the reason this can be good good is that if your code raised an error you did not expect, you now can't accidentally swallow that exception and hide a bug from yourself.

"How does that differ from putting the code after the try?"

Depending on the code, it may belong there, yes.

There are more varied reasons it may not.

TODO: write up some examples.

Common exceptions

In ≤ Py2.4, any classic class (and instatiation, e.g. a string) could be raised as an exception.

Since 2.5 (and via PEP352) exceptions inherit via the BaseException class.

It is directly extended by only a few, e.g. Exception, GeneratorExit, SystemExit, KeyboardInterrupt, StopIteration, StopAsyncIteration

All everyday built-in exceptions inherit from Exception, and most of them then via StandardException.

Exception Before 2.5 it was the base error, exceptions.Exception. Since 2.5 it inherits from BaseException

StandardError (the parent of all but a few special exceptions. This was removed in python3 as there was almost no use)
- ValueError, indicating some problem with the data handed along (and not specifically something like IndexError, LookupError)
  - UnicodeError
    - UnicodeEncodeError, UnicodeDecodeError, also UnicodeTranslateError
- ArithmeticError, mostly for:
  - ZeroDivisionError (note it may take fewer lines to add a test for zero before dividing than to catch and handle the exception)
- EnvironmentError (mostly parent to the below two)
  - IOError
  - OSError - error in the underlying system (errno-style) (cf. SystemError)
- RuntimeError
  - NotImplementedError - meant for use in relatively abstract base classes to signal the subclass should have overriden the method with a real implementation.
- AttributeError (when reference/assignment fails)
- NameError
- LookupError, the base for:
  - IndexError
  - KeyError
- SyntaxError
- SystemError - python-internal error - possibly a bug (cf. OSError)
- MemoryError
StopIteration - used in iterators. Not really an error in its common use.
GeneratorExit - used in generators. Not really an error in its common use. (inherits from BaseException since 2.6)
KeyboardInterrupt - Generated in response to Ctrl-C (inherits from BaseException since 2.5)
SystemExit used by sys.exit, and really an error (inherits from BaseException since 2.5)

(Note that this is a small selection. For a more complete tree, see the bottom of this page)

Specific errors worth noting

OSError will be a common one. You can inspect its errno (which are C-library error number(verify), see e.g. [1]).

2 (errno.ENOENT): usually 'file not found'

13 (errno.EACCESS): permission denied

12 (errno.ENOMEM): Cannot allocate memory, quite possibly in a subprocess forking (then maybe specifically virtual memory)

Technically you should test against these names, but the standard set is so constant that you can get away with hardcoding numbers)

Network-related exceptions

...and their inheritance. These are mostly the set relevant to using urllib/urllib2:

Exception (exceptions.Exception)

socket.error (inherits from IOError since py2.6, did not before that)
- ...most commonly socket.timeout
httplib.HTTPException
- httplib.BadStatusLine (parse error on the status, which is typically the first to fail on a non-HTTP response(verify))
- (...and various others, see httplib)
StandardError
- EnvironmentError
  - IOError
    - urllib2.URLError (no network connection, route, server, or some bad relatively low-level response)
      - urllib2.HTTPError (Mostly triggered by HTTP response codes between 400 and 600, which indicates request errors, proxy errors, server-side errors and such)

Notes:

various operations (open, read) can throw these

exceptions without a module come from the exceptions and act as builtins in that they are always in scope.

most real-world problems seem to be catchable with URLError and socket.timeout. When all you want to do is avoid these errors from bombing out your program, try something short yet still informative like:

except (socket.error, urllib2.URLError, httplib.HTTPException), e:
    print 'Networking problem, %s: %s'%(e.__class__, str(e))

If you catch them separately, think about order, particularly when subclassing is involved. If you want to HTTPError (as a reponse error) and URLError (as an other-network-problems error), catch them in that order.

Different exceptions attach different details, in members like reason, code. When your catch code is general, you may want to use hasattr() or similar to avoid member access problems.

Custom exceptions

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

When you write libraries, or other shared code, you regularly want to raise exceptions.

If you raise the most basic Exception, then people catching that cannot catch only that without also catching everything else.

If you raise something like ValueError, this is still partly true: if everyone does that, it's still hard to catch something specific

This is often still manageable, in that try-exception around short blocks may only ever catch a handful of different exception.

If you want to be more distinguishing

The upside is that now your client code can tell in much more detail what exceptions it's catching.

One downside is that it must entangle with your code a little, naming that specific exception inside another library. That's fine in that you are using that library anyway, but still a thorn in some people's eyes.

When you want to throw an error you can catch in a specific Except, you can declare custom exceptions.

If you only care about having it be distinctly named thing, then the simplest declaration being something like:

class MyException(Exception):
    pass

after which you an do

try:
     raise MyException('foo')
except MyException as me:
    print( 'Exception', me )

You can go crazy with all the possible errors you can think of -- yet there is a good argument in only introducing the smallest set of cases you actively expect to handle in distinct code paths.

if it's about eventually fixing bugs, then making error strings more informative may be a more valuable use of your time.

Also, note that if you do this in modules, you require people to reach inside your module to catch these.

It also means you require them to know all your errors, and the tree. This is sometimes the most correct way to handle and report errors And sometimes is a very terious, and unnecessarily tightly coupled setup.

So I personally also (ab)use ValueError, which is fine when you only care about reading what's wrong - particularly when the only sensible response is "Well, I give up on this case" anyway.

If you want to alter the way you call the exception (e.g. with extra arguments) you'll want something like:

class MyException(Exception):
    def __init__(self, message, otherstuff):
        Exception.__init__(self, message) # call base class constructor
        # Your added code:
        self.otherstuff = otherstuff

Grittier details

sys.exc_info() gives information about the exception currently being handled.

Returns a 3-tuple:

exception type - usually a class object
exception value - usually a class instance, specifically the argument to the raise
traceback object - presents the call stack at time of exception

Python used allow string arguments to raise, but doesn't do so anymore, hence the 'usually'

The function seems to search for the most recent exception, specific to the thread and the stack (frame).

In most programs, uncaught exceptions will make it to sys.excepthook [2]. In python programs the next step is exiting, in interactive python it returns to the promps.

You can replace excepthook with your own function, and various daemons (particularly multithreaded ones), frameworks and such do so.

Exceptions in threads

Notes:

When an exception happens within a thread

that thread

prints its stack trace

then quits (stops being alive, because its run() terminated

This does not affect other threads

thread.interrupt_main() raises a KeyboardInterrupt in the main thread. This is largely a convenience function that allows you to catch that Interrupt in an arbitrary thread (which is a given in some environments) and send it to some code that will handle the case. See the notes around Python notes - threading#Timely_thread_cleanup.2C_and_getting_Ctrl-C_to_work.

thread.exit() raises SystemExit, which causes the thread to silently stop (unless you manually handle it another way)

KeyboardInterrupt will not interrupt a waiting acquire. It will interrupt no sooner than the lock is acquired.

If this matters, you may want to spinlock (a while, a non-blocking acquire attempt, and a sleep call)

Warnings

Warnings are a variation on exceptions that are more flexible and less fatal.

Example:

warnings.warn("calling walk() twice is ill defined")

They act somewhat like exceptions in that they can be caught by the same mechanism (which lets you swallow specific ones you don't care about).

...but if not caught they will not stop execution - they will be fed into a function that, by default, prints them to sys.stderr

(That doesn't mean it will always behave ideally; I at one time had a problem where web serving seemed to hang, without exception or response, which turned out to the platform being confused by a warning)

You can change how the warning system behaves:

have them thrown as exceptions instead

e.g. when you want to be thorough about a production version. Or sometimes handy when unit testing.

redirect them elsewhere (often by setting the warnings.showwarning function to your own function, e.g. loggging the string instead)

filter them and/or react differently to different categories of warnings

(almost) completely ignore/hide them

They can be useful for debugging, warning you of bad things that should not make the program bork.

They are perhaps most often seen giving programmers warnings about deprecation.

Warnings you may have to deal with

DeprecationWarning: the sets module is deprecated

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

there used to be a sets module with sets.Set and sets.ImmutableSet

≤2.5, you could import the sets module.

≥2.6, set and frozenset are built-in types (also improved), and importing sets will yield:

DeprecationWarning: the sets module is deprecated

A fallback fix to tide you over, via an exception because it's shorter:

try:
    set
except NameError:
    from sets import Set as set

DeprecationWarning: The popen2 module is deprecated. Use the subprocess module

popen2 became deprecated in py2.6.

The subprocess module was introduced in 2.4, so you should probably just move to it - see Python usage notes/Subprocess#subproces_module.

Python notes - exceptions, warnings

Contents

Exceptions

args, str, and such

Catching more than one exception

try, except, and finally

...and else

Common exceptions

Specific errors worth noting

Network-related exceptions

Custom exceptions

Grittier details

Exceptions in threads

See also

Warnings

Warnings you may have to deal with

DeprecationWarning: the sets module is deprecated

DeprecationWarning: The popen2 module is deprecated. Use the subprocess module

Navigation menu