An underrated feature in Python 3(blog.ionelmc.ro)
blog.ionelmc.ro
An underrated feature in Python 3
http://blog.ionelmc.ro/2014/08/03/the-most-underrated-feature-in-python-3/
42 comments
What I don't understand about Python is why it's not printing the str() of the variables in the corresponding lines of the traceback. About 95% of all debugging starts with adding prints or logging messages before the lines mentioned in the traceback. Or am I doing something wrong here (since 5 years...)?
For one, I think repr is more fit.
And if repr (or str) has a bug, now you are chasing two issues! Besides, repr (or str) may take a lot of processing time and it is not unusual to log an exception stack trace but keep running.
At last, if you are going to debug, a lot of times is better to use a debugger instead of putting print (or log) statements all around. I prefer pydev for this. You can even attach to a python process running as a different user or even on a different computer.
I documented how to do it in a post on my blog (sorry, in Spanish). http://aurelianito.blogspot.com.ar/2013/05/debugueando-en-ro...
And if repr (or str) has a bug, now you are chasing two issues! Besides, repr (or str) may take a lot of processing time and it is not unusual to log an exception stack trace but keep running.
At last, if you are going to debug, a lot of times is better to use a debugger instead of putting print (or log) statements all around. I prefer pydev for this. You can even attach to a python process running as a different user or even on a different computer.
I documented how to do it in a post on my blog (sorry, in Spanish). http://aurelianito.blogspot.com.ar/2013/05/debugueando-en-ro...
If you prefer the console, pudb is an absolutely fantastic curses debugger. I love it.
If you __str__ is broken, the traceback would fail which would be awful.
Once I wrote a __str__ method on a class which got into an infinite loop :) When I showed the traceback to an advanced developer, he told me immediately what was happening. How could we figure out the problem if my __str__ was broken and traceback would fail?
Also recently I struggled with py.test which automatically does that (calling str() on every local variable in the test case) but it is broken, so py.test showed me strange Unicode Decode exceptions (it turned out, nothing wrong was in my code but py.test) and I had no idea what was going on. Took me hours to figure it out py.test has a issues with __future__ unicode_literals...
tl;dr: not a good idea
Once I wrote a __str__ method on a class which got into an infinite loop :) When I showed the traceback to an advanced developer, he told me immediately what was happening. How could we figure out the problem if my __str__ was broken and traceback would fail?
Also recently I struggled with py.test which automatically does that (calling str() on every local variable in the test case) but it is broken, so py.test showed me strange Unicode Decode exceptions (it turned out, nothing wrong was in my code but py.test) and I had no idea what was going on. Took me hours to figure it out py.test has a issues with __future__ unicode_literals...
tl;dr: not a good idea
Doesn't seem like a showstopper to me; just fall back to current behavior if str() fails or takes too long, and truncate if it's too verbose.
How you define "too long" and "too verbose"?
Keep a running tally of time spent stringifying. If it passes 1000 msec over the entire life of the program, disable the feature.
As for length, maybe 100 chars? I think that's around what repr() often uses.
As for length, maybe 100 chars? I think that's around what repr() often uses.
> Simple is better than complex.
I think Python exceptions are verbose enough, not necessary to make it more verbose, more complicated.
I think Python exceptions are verbose enough, not necessary to make it more verbose, more complicated.
Can str() call into a C extension that then loops forever?
Can we contrive esoteric situations where this feature would fail spectacularly? Yes.
Do they matter in reality? No.
Make it an optional feature to be toggled with an interpreter flag. People like me can turn it on, you can keep it turned off.
Do they matter in reality? No.
Make it an optional feature to be toggled with an interpreter flag. People like me can turn it on, you can keep it turned off.
That doesn't seem that esoteric to me: in general the problem is that repr() can result in execution of arbitrary code, and that there's no way to cleanly and reliably terminate arbitrary code. This includes Python code: it's possible to terminate it at an arbitrary point, but that leaves things in an indeterminate state.
I think it's pretty important that you shouldn't add features with unpredictable behaviour into the core of the language, particularly into error handling code.
I think it's pretty important that you shouldn't add features with unpredictable behaviour into the core of the language, particularly into error handling code.
It's esoteric because in reality it's the kind of bug that never happens. And if it happens it's trivial to identify the cause; "Oh, it hangs in the middle of a strack-trace, what could it possibly be...".
Why would your code contain repr()-functions that trigger "unpredictable behavior" to begin with anyway?
Why would your code contain repr()-functions that trigger "unpredictable behavior" to begin with anyway?
I typically will throw in a debugger instead of a print statement if I'm debugging locally.
import ipdb; ipdb.set_trace()
Then I'll explore the context variables and figure out what the hell is going on.
import ipdb; ipdb.set_trace()
Then I'll explore the context variables and figure out what the hell is going on.
I think 99.9999% experienced python devs do this.
Why can't PSF make it default?
At least some kind of commandline switch to break to last traceback context by default. e.g.
python --debug my_code.py
Why can't PSF make it default?
At least some kind of commandline switch to break to last traceback context by default. e.g.
python --debug my_code.py
they did: python -m pdb my_code.py
ipdb just made my day.
Thanks.
Thanks.
This is a great idea, although repr() would probably be better than str().
I've spent a lot of time debugging Python over the years. This honestly never occured to me until you mentioned it.
Edit:
Everyone else jumped in before me. I think it would be useful as a command line switch to python to turn on the behavior (maybe call it extended exceptions).
Python would need to run the extended exception generation under a try/except that fell back to regular exceptions if an exception was raised while calling repr()
I've spent a lot of time debugging Python over the years. This honestly never occured to me until you mentioned it.
Edit:
Everyone else jumped in before me. I think it would be useful as a command line switch to python to turn on the behavior (maybe call it extended exceptions).
Python would need to run the extended exception generation under a try/except that fell back to regular exceptions if an exception was raised while calling repr()
How about automatically giving you a repl context at the exception and letting you move up and down the call stack.
http://werkzeug.pocoo.org/docs/debug/#using-the-debugger
http://werkzeug.pocoo.org/docs/debug/#using-the-debugger
The str or repr of a variable might result in megabytes of output (or worse, calling it might have a side effect). Still, it'd be nice to at least show the first 100 bytes or so.
The string representation (repr() would be a better fit actually) and the amount of local variables could be enormous.
To be honest. As cool as this feature is in the 1% of cases where I wanted it, so annoying is it in 99% of all other cases. Tracebacks on Python 3 became a lot more complex because very often exceptions are intentionally swallowed to be reraised differently and they just mess up the traceback now.
For instance any custom collection that acts as a decorator will now generate very large tracebacks.
I would have much preferred if the reraising with old traceback would have been a feature you can enable per site where you raise.
For instance any custom collection that acts as a decorator will now generate very large tracebacks.
I would have much preferred if the reraising with old traceback would have been a feature you can enable per site where you raise.
I don't understand what are you talking about. AFAIK, you don't have to reraise the previous exception:
In [1]: try:
...: raise Exception('Something is wrong')
...: except Exception as e:
...: raise
...:
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
<ipython-input-1-ef6cea2fccac> in <module>()
1 try:
----> 2 raise Exception('Something is wrong')
3 except Exception as e:
4 raise
5
Exception: Something is wrong
In [2]: try:
raise Exception('Something is wrong with RERAISE')
except Exception as e:
raise e
...:
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
<ipython-input-2-c6729aaebd3d> in <module>()
2 raise Exception('Something is wrong')
3 except Exception as e:
----> 4 raise e
5
<ipython-input-2-c6729aaebd3d> in <module>()
1 try:
----> 2 raise Exception('Something is wrong with RERAISE')
3 except Exception as e:
4 raise e
5
Exception: Something is wrong with RERAISE> I don't understand what are you talking about. AFAIK, you don't have to reraise the previous exception:
Correct, and also you can explicitly suppress the other traceback now (3.2+)
I just wish it was the default unless you opted in. If I catch down an exception and raise another one I do not want the old one. Only in very rare cases am I interested in what lead to it. I have lots of hacks in Jinja2 now to hide useless tracebacks that would only confuse users.
For instance a jinja context tries different dicts through a chain in __getitem__. The naive implementation on Python 3 generates two nearly equivalent tracebacks that are very confusing.
Correct, and also you can explicitly suppress the other traceback now (3.2+)
I just wish it was the default unless you opted in. If I catch down an exception and raise another one I do not want the old one. Only in very rare cases am I interested in what lead to it. I have lots of hacks in Jinja2 now to hide useless tracebacks that would only confuse users.
For instance a jinja context tries different dicts through a chain in __getitem__. The naive implementation on Python 3 generates two nearly equivalent tracebacks that are very confusing.
I don't think Jinja is a relevant example here - it's not something people usually write. While you take great care about the API and error handling most other people are sloppy and their code is hard to debug because they unknowingly cripple the tracebacks. That's why I like this feature just the way it is (an opt out, instead of explicit opt in) - cause it needs to cater for the 99% of the people who didn't even bother to reraise properly in Python 2.
If you don't want the older exception, why are you giving it to the constructor of the new exception?
Just raise SomeErrorType("error message"), and it will not polute your traceback. Raising SomeErrorType(anotherError) is there for when you care about anotherError.
Just raise SomeErrorType("error message"), and it will not polute your traceback. Raising SomeErrorType(anotherError) is there for when you care about anotherError.
> If you don't want the older exception, why are you giving it to the constructor of the new exception?
Exceptions are chained automatically through the stracktrace and not because they are passed to the constructor. By raising an exception from an exception handler you automatically chain them.
Exceptions are chained automatically through the stracktrace and not because they are passed to the constructor. By raising an exception from an exception handler you automatically chain them.
I don't understand Python exceptions that well. But it's great to have the pattern of maintaining the stack trace when re-raising an exception. Thanks OP!
(In case anyone missed it)
(In case anyone missed it)
except foo.FooException as e:
raise BarException, BarException(e), sys.exc_info()[2]Had the pleasure of working with the author (Ionel) a few years back on some Django projects. Always love reading his Python posts. Really inspiring developer.
Actually passing the exception is not required. Even if "e" is not an argument to BarException, the behaviour is the same.
Why is the current title ("An underrated feature in Python 3") neither the original source title ("The most underrated feature in Python 3") or a title more descriptive of the content (e.g., "Exception chaining in Python 3").
If we're going to get title changes, can we at least get useful title changes?
If we're going to get title changes, can we at least get useful title changes?
>If we're going to get title changes, can we at least get useful title changes?
No. If the title changes were useful, what would we discuss on HN?
No. If the title changes were useful, what would we discuss on HN?
Can a moderator de-hype the title, please?
[Edit: thanks!]
[Edit: thanks!]
I don't understand why sometimes the submitted title must slavishly stick to the original title - otherwise it is considered editorializing - and why sometimes it's OK to change it.
In this case, "The most underrated feature in Python 3" was a fine title. It brought across the excitement of the post's author for this particular feature. I agree, it is a bit link-baity, but a much better version would be "Chaining exceptions is the most underrated feature in Python 3".
The current version ("An underrated feature in Python 3") is extremely nondescript, and I wouldn't have clicked it. It's almost like "A feature in Python 3".
If you object to the use of "most"... the author is obviously using hyperbole. There is no objective measure of "underratedness" of course. I wish people would not take everything so literally, and try to interpret a bit what they read instead. What the author wanted to say was probably (and this is an equally good title): "I really like that Python 3 has chained exceptions!"
In this case, "The most underrated feature in Python 3" was a fine title. It brought across the excitement of the post's author for this particular feature. I agree, it is a bit link-baity, but a much better version would be "Chaining exceptions is the most underrated feature in Python 3".
The current version ("An underrated feature in Python 3") is extremely nondescript, and I wouldn't have clicked it. It's almost like "A feature in Python 3".
If you object to the use of "most"... the author is obviously using hyperbole. There is no objective measure of "underratedness" of course. I wish people would not take everything so literally, and try to interpret a bit what they read instead. What the author wanted to say was probably (and this is an equally good title): "I really like that Python 3 has chained exceptions!"
What title would you like ? What's exactly hyped ?
"The most underrated feature" is total hype. It's exactly the sort of thing the Guidelines link at the bottom of every HN page says to remove.
Maybe, "Python 3 exception-handling improvement."
Maybe, "Python 3 exception-handling improvement."
Hmm, it's just a figure of speach. "The most underrated feature" -> "I really like this feature and think it deserves more attention."
Just like "the best programming language ever" -> "a programming language I am really excited about, that fits my needs very well".
The hyperbole is not the problem IMO, it's the nondescriptiveness. I think the title should have something about exception handling in it.
The enforcement of the Guidelines has been pretty arbitrary, and quite often good (editorialized) titles have been turned back into something nondescript. Or now the other way around, the original title is editorialized to be less interesting. I think the most important criteria for a title are: does it get across the topic of the link, and the target page author's intention?
Just like "the best programming language ever" -> "a programming language I am really excited about, that fits my needs very well".
The hyperbole is not the problem IMO, it's the nondescriptiveness. I think the title should have something about exception handling in it.
The enforcement of the Guidelines has been pretty arbitrary, and quite often good (editorialized) titles have been turned back into something nondescript. Or now the other way around, the original title is editorialized to be less interesting. I think the most important criteria for a title are: does it get across the topic of the link, and the target page author's intention?
Implicit Exception Chaining in Python 3.0