Python notes - exceptions, warnings

From Helpful
Jump to: navigation, search
Syntaxish: syntax and language · importing, modules, packages · iterable stuff · concurrency

IO: networking and web · filesystem

Data: Numpy, scipy · pandas · struct, buffer, array, bytes, memoryview · Python database notes

Image, Visualization: PIL · Matplotlib, pylab · seaborn · bokeh · plotly


Processes: threading · subprocess · multiprocessing · joblib · pty and pexpect

Stringy: strings, unicode, encodings · regexp · command line argument parsing · XML

date and time

semi-sorted

Exceptions

args, str, and such

Exceptions have an
args
variable, which is a tuple.
Often with a single string, sometimes more data-like details.
is set via
BaseException.__init__(*args)
read out by
BaseException.__str__
use/meaning is mostly up to you.


It makes it easy to put a message and/or arbitrary values in there with minimal work:

class MyException(Exception):
    pass
 
try:
    raise MyException("bad thing",1.9)
except Exception, e:
    print e.args
    print repr(e)
    print str(e)


Specific exceptions may add some more structured details elsewhere. For example, OSError adds errno.

Be careful when assuming what's in args. It's probably good style to only look for values when catching very specific exceptions - and note that in some cases, names/details have changed over time.



Printing an exception:

  • repr() is most informative, though sometimes a little clunky.
  • str() mostly just prints args (and taking out some brackets when len(args)==1)
  • many people put a readable thing in args[0]
  • e.message (on BaseException) was introduced and then deprecated
don't assume it's there, don't use it


https://docs.python.org/2/library/exceptions.html

https://wiki.python.org/moin/HandlingExceptions

https://julien.danjou.info/blog/2016/python-exceptions-guide -->

Catching more than one exception

As you probably know, catching distinct exceptions is not a special case:

except socket.error, e:
    print 'Socket error: %s'%str(e)
except httplib.HTTPException:
    print 'HTTP problem.'


In some situations, you want to react the same way, e.g. catching multiple network-related exceptions and just say "network was stupid" message.

What you are looking for is:

except (socket.error, urllib2.URLError, httplib.HTTPException), e:
    # when you want to log the error, you may want to 
    # mention the exception type as well as its arguments, so:
    print 'Networking problem, %s: %s'%(e.__class__, str(e))


And not:

except Exception1, Exception2:

This is actually just the earlier form, and means "if Exception1 is caught, bind Exception2 as a variable name in local scope that refers to the exception object" - also meaning Exception2 is never caught by this.


Since python 2.6 you can use
as
instead
(and py3 accepts only
as
), which avoids the last ambiguity, and can read a little easier:
except (Exception1, Exception2) as e:

try, except, and finally

The try and except, as a pattern, is generally known.

finally is also interesting: you can specify a code block that is always executed, including when the containing code returns, excepts, or uses continue or break.

Finally is a useful way to do cleanup that is not conditional, both in terms of execution and code cleanliness. You only have to put it in your code once, not on each separate path out of a block (or, regularly, function).


In ≤py2.4 you had to end a try block with either except or finally, so if you wanted to use finally for cleanup, for example for database transactions and/or connections, you might have written:

try:
    dbconn = connect()
    try:
        #your code, probably followed by a
        dbconn.commit()
    except SomeError:
        dbconn.rollback()
finally: #cleanup, such as
    dbconn.close()

From python 2.5 on, you can write:

try:
    #your code, probably followed by a
    dbconn.commit()
except SomeError:
    dbconn.rollback()
finally:
    dbconn.close()


Common exceptions

In ≤ Py2.4, any classic class (and instatiation, e.g. a string) could be raised as an exception.

Since 2.5 (and via PEP352) exceptions inherit via the BaseException class.

It is directly extended by only fourish: Exception, GeneratorExit, SystemExit, KeyboardInterrupt, StopIteration, which allows the latter four to be easier exception
All everyday built-in exceptions inherit from Exception, and most of them then via StandardException.


Exception Before 2.5 it was the base error, exceptions.Exception. Since 2.5 it inherits from BaseException

  • StandardError (the parent of all but a few special exceptions)
    • ValueError, indicating some problem with the data handed along (and not specifically something like IndexError, LookupError)
      • UnicodeError
        • UnicodeEncodeError, UnicodeDecodeError, also UnicodeTranslateError
    • ArithmeticError, mostly for:
      • ZeroDivisionError (note it may take fewer lines to add a test for zero before dividing than to catch and handle the exception)
    • EnvironmentError (mostly parent to the below two)
      • IOError
      • OSError - error in the underlying system (errno-style) (cf. SystemError)
    • RuntimeError
      • NotImplementedError - meant for use in relatively abstract base classes to signal the subclass should have overriden the method with a real implementation.
    • AttributeError (when reference/assignment fails)
    • NameError
    • LookupError, the base for:
      • IndexError
      • KeyError
    • SyntaxError
    • SystemError - python-internal error - possibly a bug (cf. OSError)
    • MemoryError
  • StopIteration - used in iterators. Not really an error in its common use.
  • GeneratorExit - used in generators. Not really an error in its common use. (inherits from BaseException since 2.6)
  • KeyboardInterrupt - Generated in response to Ctrl-C (inherits from BaseException since 2.5)
  • SystemExit used by sys.exit, and really an error (inherits from BaseException since 2.5)

(Note that this is a small selection. For a more complete tree, see the bottom of this page)

Specific errors worth noting

OSError will be a common one. You can inspect its errno (which are the OS's errno, see e.g. [1]), which technically you should test against these constants but in practice are constants:

  • 13 (errno.EACCESS): permission denied
  • 2 (errno.ENOENT): usually 'file not found'
  • 12 (errno.ENOMEM): Cannot allocate memory, quite possibly in a subprocess forking (then maybe specifically virtual memory)


Network-related exceptions

...and their inheritance. These are mostly the set relevant to using urllib/urllib2:

Exception (exceptions.Exception)

  • socket.error (inherits from IOError since py2.6, did not before that)
    • ...most commonly socket.timeout
  • httplib.HTTPException
    • httplib.BadStatusLine (parse error on the status, which is typically the first to fail on a non-HTTP response(verify))
    • (...and various others, see httplib)
  • StandardError
    • EnvironmentError
      • IOError
        • urllib2.URLError (no network connection, route, server, or some bad relatively low-level response)
          • urllib2.HTTPError (Mostly triggered by HTTP response codes between 400 and 600, which indicates request errors, proxy errors, server-side errors and such)

Notes:

  • various operations (open, read) can throw these
  • exceptions without a module come from the exceptions and act as builtins in that they are always in scope.
  • most real-world problems seem to be catchable with URLError and socket.timeout. When all you want to do is avoid these errors from bombing out your program, try something short yet still informative like:
except (socket.error, urllib2.URLError, httplib.HTTPException), e:
    print 'Networking problem, %s: %s'%(e.__class__, str(e))
  • If you catch them separately, think about order, particularly when subclassing is involved. If you want to HTTPError (as a reponse error) and URLError (as an other-network-problems error), catch them in that order.
  • Different exceptions attach different details, in members like reason, code. When your catch code is general, you may want to use hasattr() or similar to avoid member access problems.

Custom exceptions

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

I personally mostly (ab)use ValueError for most things, which is fine when you only care about reading what's wrong.


It's easy enough to make custom exceptions, the simplest declaration being something like:

class MyException(Exception):
    pass

You can go crazy and make your own tree of them.

There is however good argument that you should only add what you would handle in distinct code paths.

If not, e.g. you're going to bork out anyway, making the error strings more informative is more valuable use of your time.



If you want to alter the way you call the exception (e.g. with extra arguments) you'll want something like:

class MyException(Exception):
    def __init__(self, message, otherstuff):
        Exception.__init__(self, message) # call base class constructor
        # Your added code:
        self.otherstuff = otherstuff

Grittier details

sys.exc_info() gives information about the exception currently being handled.

Returns a 3-tuple:

  • exception type - usually a class object
  • exception value - usually a class instance, specifically the argument to the raise
  • traceback object - presents the call stack at time of exception

Python used allow string arguments to raise, but doesn't do so anymore, hence the 'usually'

The function seems to search for the most recent exception, specific to the thread and the stack (frame).



In most programs, uncaught exceptions will make it to sys.excepthook [2]. In python programs the next step is exiting, in interactive python it returns to the promps.

You can replace excepthook with your own function, and various daemons (particularly multithreaded ones), frameworks and such do so.


Exceptions in threads

Notes:

  • When an exception happens within a thread
that thread
prints its stack trace
then quits (stops being alive, because its run() terminated
This does not affect other threads


  • thread.exit() raises SystemExit, which causes the thread to silently stop (unless you manually handle it another way)


  • KeyboardInterrupt will not interrupt a waiting acquire. It will interrupt no sooner than the lock is acquired.
If this matters, you may want to spinlock (a while, a non-blocking acquire attempt, and a sleep call)


See also:

See also

Warnings

Warnings are a variation on exceptions that are more flexible and less fatal.


They act somewhat like exceptions in that they can be caught by the same mechanism (which lets you swallow specific ones you don't care about).

...but if not caught they will not stop execution - they will be fed into a function that, by default, prints them to sys.stderr


(That doesn't mean it will always behave ideally; I at one time had a problem where web serving seemed to hang, without exception or response, which turned out to the platform being confused by a warning)


You can change how the warning system behaves:

  • have them thrown as exceptions instead
e.g. when you want to be thorough about a production version. Or sometimes handy when unit testing.
  • redirect them elsewhere (often by setting the warnings.showwarning function to your own function, e.g. loggging the string instead)
  • (almost) completely ignore/hide them


They can be useful for debugging, warning you of bad things that should not make the program bork.

They are perhaps most often seen giving programmers warnings about deprecation.


See also


Warnings you may have to deal with

DeprecationWarning: the sets module is deprecated

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

there used to be a sets module with sets.Set and sets.ImmutableSet

≤2.5, you could import the sets module.

≥2.6,
set
and
frozenset
are built-in types (also improved), and importing sets will yield:
DeprecationWarning: the sets module is deprecated


A fallback fix to tide you over, via an exception because it's shorter:

try:
    set
except NameError:
    from sets import Set as set

DeprecationWarning: The popen2 module is deprecated. Use the subprocess module

popen2 became deprecated in py2.6.

The subprocess module was introduced in 2.4, so you should probably just move to it - see Python usage notes/Subprocess#subproces_module.