Python notes - exceptions, warnings
| Syntaxish: syntax and language · type stuff · changes and py2/3 · decorators · importing, modules, packages · iterable stuff · concurrency · exceptions, warnings
IO: networking and web · filesystem Data: Numpy, scipy · pandas, dask · struct, buffer, array, bytes, memoryview · Python database notes Image, Visualization: PIL · Matplotlib, pylab · seaborn · bokeh · plotly
Stringy: strings, unicode, encodings · regexp · command line argument parsing · XML speed, memory, debugging, profiling · Python extensions · semi-sorted |
Exceptions
Exception string and details
- Generic exceptions are not guaranteed to set anything on the exception object, so the most generic way to get the error string seems to be str(exception)
- A number of exceptions have an args variable., which should be a tuple.
- use/meaning within that tuple is up to who writes that exception's code
- many people put a readable thing in args[0]
- Sometimes it contains just that single string, sometimes more data-like details.
- ...and for specific exceptions where you know how it is used, the last can be very useful
- args will often be set via BaseException.__init__(*args)
- ...which is what makes makes it easy to put a message and/or arbitrary values in there with minimal work
- BaseException.__str__ will often be reading it out
- A few exceptions may add some more structured details elsewhere.
- For example, OSError adds errno[1] when relevant -- but not always!
- e.message (on BaseException) was introduced a while back, and later deprecated
- don't assume it's there, don't use it
class MyException(Exception):
pass
try:
raise MyException("bad thing",1.9)
except Exception as e:
print( e.args ) # the tuple
print( repr(e) ) # does .__class__.__name__ stuff too
print( str(e) ) # str() of the tuple
which outputs
('bad thing', 1.9)
MyException('bad thing', 1.9)
('bad thing', 1.9)
So when printing an exception:
- repr() is most informative, though sometimes a little clunky.
- str() is mostly just args (and taking out some brackets when len(args)==1)
Be careful when assuming what's in args.
It's probably good style to only look for values when catching very specific exceptions - and note that in some cases, names/details have changed over time.
https://docs.python.org/2/library/exceptions.html
https://wiki.python.org/moin/HandlingExceptions
https://julien.danjou.info/blog/2016/python-exceptions-guide -->
raise from
More technically, raise from has two variants:
- if the expression after from is an exception instance, it will be attached to the raised exception as __cause__
- if the expression after from is an exception class, it will be instantiated and set on __cause__
...but why?
Usually to give a more readable context/reasoning to a generic exception.
Sometimes instead of the original -- however, this easily amounts to hiding the error, and consider that usually, only programmers who need to fix the offending code are really reading your exception
Consider:
# some code that amounts to 0//0 because you did something like sum(cases)/len(cases)
will give you
ZeroDivisionError: integer division or modulo by zero
Technically correct, but the traceback will not tell you why.
If you expect that, and do:
try:
# some code that amounts to 0//0 because you did something like sum(cases)/len(cases)
except ZeroDivisionError as exc:
raise ValueError("statistics failed because you handed in no cases") from exc
Will show a traceback showing both exceptions (and you can fish out __cause__ in code), e.g. like:
Traceback (most recent call last): File "<stdin>", line 2, in <module> ZeroDivisionError: integer division or modulo by zero The above exception was the direct cause of the following exception: Traceback (most recent call last): File "<stdin>", line 4, in <module> ValueError: statistics failed because you handed in no cases
Note that such a two-exceptions-mentioned happens anyway
when a new exception happens inside an except block (when you raise anything but the exception you are handling(verify)),
so something like:
try:
# some code that amounts to 0//0 because you did something like sum(cases)/len(cases)
except:
raise ValueError("statistics failed because you handed in no cases")
also works, but is more implicit behaviour -- the from makes it clearer what will happen ()
Note also that you can suppress the original exception using from None
https://docs.python.org/3/reference/simple_stmts.html#raise
Catching more than one exception
As you probably know, catching distinct exceptions is not a special case:
except socket.error as e: # py3; in py2 this was except socket.error as e:
print 'Socket error: %s'%str(e)
except httplib.HTTPException:
print 'HTTP problem.'
In some situations, you want to react the same way, e.g. catching multiple network-related exceptions and just say "network was stupid" message.
What you are looking for is:
except (socket.error, urllib2.URLError, httplib.HTTPException) as e:
# when you want to log the error, you may want to
# mention the exception type as well as its arguments, so:
print 'Networking problem, %s: %s'%(e.__class__, str(e))
And not:
except Exception1, Exception2:
The py3 move to use 'as' instead of a comma was in part to avoid this actually being the first form in disguise, meaning"if Exception1 is caught, bind Exception2 as a local variable name that refers to the exception object" - also meaning Exception2 is never caught by this.
Since python 2.6 you can use as instead, since py3 you can use only as),
which avoids the last ambiguity, and can read a little easier:
try, except, and finally
The try and except, as a pattern, is generally known.
finally lets you specify a code block that is always executed,
- including when the containing code returns
- including when the containing code excepts
- including when the containing code uses continue or break.
As such, finally is a useful way (in terms of execution and code cleanliness) to do cleanup that is not conditional. You only have to put it in your code once, not on each separate path out of a block (or, regularly, function).
In ≤py2.4 you had to end a try block with either except or finally, so if you wanted to use finally for cleanup, for example for database transactions and/or connections, you might have written:
try:
dbconn = connect()
try:
#your code, probably followed by a
dbconn.commit()
except SomeError:
dbconn.rollback()
finally: #cleanup, such as
dbconn.close()
From python 2.5 on, you can write:
try:
#your code, probably followed by a
dbconn.commit()
except SomeError:
dbconn.rollback()
finally:
dbconn.close()
...and else
try:
# code
except:
# executed if code raised an exception
else:
# executed if try block finishes without exception
finally:
# executed regardless of exception
The code you put in the else block is executed only if execution flows out of the try block naturally, so it will not execute if there was an exception, return, continue, or break.
...so it differs from finally in that
- Finally still executes if there was an return, continue, or break, and regardless of whether there was an exception or not
- else executes only when none of those happened.
But perhaps more importantly, it differs from the main code block in that
- exceptions in the else block is not caught by the except.
What is this good for?
I have seen a lot of vague and/or poor examples, where it's just different rather than clearly better.
Most of the decent examples I've seen involve exception handling - better isolating what you're handling. And yes, those examples can be written equivalently with just try-excepts-finally.
Some examples also make a decent case that the blocks and flow might stay a little more compact and/or readable.
"How does that differ from putting the code inside the try?"
- Depending on what the code is, it may belong there, yes.
- The exception-y reason is that the try block can trip into the except blocks, the else block cannot, and the reason this can be good good is that if your code raised an error you did not expect, you now can't accidentally swallow that exception and hide a bug from yourself.
"How does that differ from putting the code after the try?"
- Depending on the code, it may belong there, yes.
- There are more varied reasons it may not.
TODO: write up some examples.
Common exceptions
In ≤ Py2.4, any classic class (and instatiation, e.g. a string) could be raised as an exception.
Since 2.5 (and via PEP352) exceptions inherit via the BaseException class.
- It is directly extended by only a few, e.g. Exception, GeneratorExit, SystemExit, KeyboardInterrupt, StopIteration, StopAsyncIteration
- All everyday built-in exceptions inherit from Exception, and most of them then via StandardException.
Exception (Before 2.5 this was the base error, exceptions.Exception. Since 2.5 it inherits from BaseException)
- StandardError (the parent of all but a few special exceptions. This was removed in python3 as there was almost no use)
- ValueError, indicating some problem with the data handed along (and not specifically something like IndexError, LookupError)
- UnicodeError
- UnicodeEncodeError, UnicodeDecodeError, also UnicodeTranslateError
- UnicodeError
- ArithmeticError, mostly for:
- ZeroDivisionError (note it may take fewer lines to add a test for zero before dividing than to catch and handle the exception)
- EnvironmentError (mostly parent to the below two)
- IOError
- OSError - error in the underlying system (errno-style) (cf. SystemError)
- RuntimeError
- NotImplementedError - meant for use in relatively abstract base classes to signal the subclass should have overriden the method with a real implementation.
- AttributeError (when reference/assignment fails)
- NameError
- LookupError, the base for:
- IndexError
- KeyError
- SyntaxError
- SystemError - python-internal error - possibly a bug (cf. OSError)
- MemoryError
- ValueError, indicating some problem with the data handed along (and not specifically something like IndexError, LookupError)
- StopIteration - used in iterators. Not really an error in its common use.
- GeneratorExit - used in generators. Not really an error in its common use. (inherits from BaseException since 2.6)
- KeyboardInterrupt - Generated in response to Ctrl-C (inherits from BaseException since 2.5)
- SystemExit used by sys.exit, and really an error (inherits from BaseException since 2.5)
(Note that this is a small selection. For a more complete tree, see the bottom of this page)
Specific errors worth noting
OSError will be a common one. You can inspect its errno (which are C-library error number(verify), see e.g. [2]).
- 2 (errno.ENOENT): usually 'file not found'
- 13 (errno.EACCESS): permission denied
- 12 (errno.ENOMEM): Cannot allocate memory, quite possibly in a subprocess forking (then maybe specifically virtual memory)
Technically you should test against these names, but the standard set is so constant that you can get away with hardcoding numbers)
...and their inheritance. These are mostly the set relevant to using urllib/urllib2:
Exception (exceptions.Exception)
- socket.error (inherits from IOError since py2.6, did not before that)
- ...most commonly socket.timeout
- httplib.HTTPException
- StandardError
- EnvironmentError
- IOError
- urllib2.URLError (no network connection, route, server, or some bad relatively low-level response)
- urllib2.HTTPError (Mostly triggered by HTTP response codes between 400 and 600, which indicates request errors, proxy errors, server-side errors and such)
- urllib2.URLError (no network connection, route, server, or some bad relatively low-level response)
- IOError
- EnvironmentError
Notes:
- various operations (open, read) can throw these
- exceptions without a module come from the exceptions and act as builtins in that they are always in scope.
- most real-world problems seem to be catchable with URLError and socket.timeout. When all you want to do is avoid these errors from bombing out your program, try something short yet still informative like:
except (socket.error, urllib2.URLError, httplib.HTTPException), e: # note that this is py2 syntax; TODO: add py3 equivalent
print 'Networking problem, %s: %s'%(e.__class__, str(e))
- If you catch them separately, think about order, particularly when subclassing is involved. If you want to HTTPError (as a reponse error) and URLError (as an other-network-problems error), catch them in that order.
- Different exceptions attach different details, in members like reason, code. When your catch code is general, you may want to use hasattr() or similar to avoid member access problems.
Custom exceptions
When you write libraries, or other shared code, you regularly want to raise exceptions.
If you raise the most basic Exception, then people catching that cannot catch only that without also catching everything else.
If you raise something like ValueError, this is still partly true: if everyone does that, it's still hard to catch something specific
- This is often still manageable, in that try-exception around short blocks may only ever catch a handful of different exception.
If you want to be more distinguishing
The upside is that now your client code can tell in much more detail what exceptions it's catching.
One downside is that it must entangle with your code a little, naming that specific exception inside another library. That's fine in that you are using that library anyway, but still a thorn in some people's eyes.
When you want to throw an error you can catch in a specific Except, you can declare custom exceptions.
If you only care about having it be distinctly named thing, then the simplest declaration being something like:
class MyException(Exception):
pass
after which you an do
try:
raise MyException('foo')
except MyException as me:
print( 'Exception', me )
You can go crazy with all the possible errors you can think of -- yet there is a good argument in only introducing the smallest set of cases you actively expect to handle in distinct code paths.
if it's about eventually fixing bugs, then making error strings more informative may be a more valuable use of your time.
Also, note that if you do this in modules, you require people to reach inside your module to catch these.
It also means you require them to know all your errors, and the tree. This is sometimes the most correct way to handle and report errors And sometimes is a very terious, and unnecessarily tightly coupled setup.
So I personally also (ab)use ValueError, which is fine when you only care about reading what's wrong - particularly when the only sensible response is "Well, I give up on this case" anyway.
If you want to alter the way you call the exception (e.g. with extra arguments) you'll want something like:
class MyException(Exception):
def __init__(self, message, otherstuff):
Exception.__init__(self, message) # call base class constructor
# Your added code:
self.otherstuff = otherstuff
Grittier details
sys.exc_info() gives information about the exception currently being handled.
Returns a 3-tuple:
- exception type - usually a class object
- exception value - usually a class instance, specifically the argument to the raise
- traceback object - presents the call stack at time of exception
Python used allow string arguments to raise, but doesn't do so anymore, hence the 'usually'
The function seems to search for the most recent exception, specific to the thread and the stack (frame).
In most programs, uncaught exceptions will make it to sys.excepthook [3]. In python programs the next step is exiting, in interactive python it returns to the promps.
You can replace excepthook with your own function, and various daemons (particularly multithreaded ones), frameworks and such do so.
Exceptions in threads
Notes:
- When an exception happens within a thread
- that thread
- prints its stack trace
- then quits (stops being alive, because its run() terminated
- This does not affect other threads
- thread.interrupt_main() raises a KeyboardInterrupt in the main thread. This is largely a convenience function that allows you to catch that Interrupt in an arbitrary thread (which is a given in some environments) and send it to some code that will handle the case. See the notes around Python notes - threading#Timely_thread_cleanup.2C_and_getting_Ctrl-C_to_work.
- thread.exit() raises SystemExit, which causes the thread to silently stop (unless you manually handle it another way)
- KeyboardInterrupt will not interrupt a waiting acquire. It will interrupt no sooner than the lock is acquired.
- If this matters, you may want to spinlock (a while, a non-blocking acquire attempt, and a sleep call)
See also:
- http://docs.python.org/library/thread.html
- http://www.voidspace.org.uk/ironpython/threading.shtml
- http://linuxgazette.net/107/pai.html
See also
Warnings
Warnings are a variation on exceptions that are more flexible and less fatal.
Example:
warnings.warn("calling walk() twice is ill defined")
They act somewhat like exceptions in that they can be caught by the same mechanism (which lets you swallow specific ones you don't care about).
...but if not caught they will not stop execution - they will be fed into a function that, by default, prints them to sys.stderr
(That doesn't mean it will always behave ideally; I at one time had a problem where web serving seemed to hang, without exception or response, which turned out to the platform being confused by a warning)
You can change how the warning system behaves:
- have them thrown as exceptions instead
- e.g. when you want to be thorough about a production version. Or sometimes handy when unit testing.
- redirect them elsewhere (often by setting the warnings.showwarning function to your own function, e.g. loggging the string instead)
- filter them and/or react differently to different categories of warnings
- (almost) completely ignore/hide them
They can be useful for debugging, warning you of bad things that should not make the program bork.
They are perhaps most often seen giving programmers warnings about deprecation.
See also
- http://docs.python.org/lib/module-warnings.html (Warnings module)
- http://docs.python.org/lib/warning-functions.html (Available functions)
- http://docs.python.org/lib/warning-filter.html (Warning filtering)
- http://docs.python.org/lib/warning-categories.html (different types of warnings)
Ignoring warnings
...temporarily:
Warnings you may have to deal with
DeprecationWarning: the sets module is deprecated
there used to be a sets module with sets.Set and sets.ImmutableSet
≤2.5, you could import the sets module.
≥2.6, set and frozenset are built-in types (also improved), and importing sets will yield:
DeprecationWarning: the sets module is deprecated
A fallback fix to tide you over, via an exception because it's shorter:
try:
set
except NameError:
from sets import Set as set
DeprecationWarning: The popen2 module is deprecated. Use the subprocess module
popen2 became deprecated in py2.6.
The subprocess module was introduced in 2.4, so you should probably just move to it.