Wield Python's super() like a Jedi(rhettinger.wordpress.com)
rhettinger.wordpress.com
Wield Python's super() like a Jedi
http://rhettinger.wordpress.com/2011/05/26/super-considered-super/
15 comments
super() supports arguments which allow you to specify which class's implementation to lookup, which is an explicit way to resolve ambiguities. Although well-designed multiple-inheritance should try to avoid those ambiguities in the first place -- a class shouldn't inherit from multiple bases with conflicting implementations of a method.
I think in a case like the one you describe, where the direct ancestor's implementation is circumvented to access a different ancestor's implementation, it should be done by calling
I think in a case like the one you describe, where the direct ancestor's implementation is circumvented to access a different ancestor's implementation, it should be done by calling
super(Ancestor).someMethod()
I disagree, that's extremely fragile. It might work for the particular class you are implementing. However, it might cause subclasses of that class to break, since those subclasses will have a different MRO than the current class, and you could end up skipping more of the MRO than you intended.
[deleted]
Personally I'm not in favor of enforcing such strict conventions, especially in a dynamic language like Python. Also isn't this pattern he repeats throughout:
class Shape:
def __init__(self, **kwds):
self.shapename = kwds.pop('shapename')
super().__init__(**kwds)
class Shape:
def __init__(self, shapename=None, **kwds):
self.shapename = shapename
super().__init__(**kwds)
These two aren't equivalent, the first example requires the shapename keyword arg, raising a KeyError when it wasn't specified
That's true. The second one might be better written as:
def __init__(self, shapename, **kwds):
def __init__(self, *, shapename, **kwds):
def __init__(self, shapename, **kwds):The version that I was rewriting:
def __init__(self, shapename=None, **kwds):
Haha, the author is one of the main contributors to python, not some high-schooler.
kwds.pop('shapename', None)
The issue with your second version is
In [1]: def init(shapename=None, **kwargs):
...: pass
...:
In [2]: init(shapename='circle', **{'shapename': 'circle'})
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
/home/teh/Desktop/<ipython console> in <module>()
TypeError: init() got multiple values for keyword argument 'shapename'That's entirely the caller's fault, not the function's. Compare:
>>> def init(**kwargs):
... pass
...
>>> init(shapename='circle', **{'shapename': 'circle'})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: init() got multiple values for keyword argument 'shapename'It seems to me that the example of combining built-in dictionary classes is naively optimistic. For starters, OrderedDict, as it happens, does not use super! It calls the dict super-class methods directly. Since dict happens to be the next class in the MRO, this doesn't really matter for the purpose of this example, but I can envision a scenario where some plucky programmer inherits from both OrderedCounter and some other dict subclass, and the result doesn't work because OrderedDict does the wrong thing.
And OrderedDict isn't the only one. Maybe for some reason I would like to have an OrderedCounter where all the counts default to 42. So I do this:
The article goes into "How to Incorporate a Non-cooperative Class", which basically says "wrap it up in a proxy class". But that's not really going to work here, since the result would be two separate dicts, with the defaultdictwrapper methods operating on one dict, and the other methods operating on the other.
And OrderedDict isn't the only one. Maybe for some reason I would like to have an OrderedCounter where all the counts default to 42. So I do this:
class DefaultOrderedCounter(defaultdict, OrderedCounter):
pass
doc = DefaultOrderedCounter(lambda: 42)
doc.update('abracadabra')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "c:\python32\lib\collections.py", line 507, in update
_count_elements(self, iterable)
File "c:\python32\lib\collections.py", line 63, in __setitem__
self.__map[key] = link = Link()
AttributeError: 'DefaultOrderedCounter' object has no attribute '_OrderedDict__map'
The article goes into "How to Incorporate a Non-cooperative Class", which basically says "wrap it up in a proxy class". But that's not really going to work here, since the result would be two separate dicts, with the defaultdictwrapper methods operating on one dict, and the other methods operating on the other.
Okay, I give. What's the markup for posting code snippets on this site?
Put two spaces at the beginning of a line for that line to be displayed monospaced.
Examplewhy would you have to subclass DefaultOrderedCounter again in order to add in the missing method?
Though I agree there's something annoying going on there with the multiple inheritence.
Though I agree there's something annoying going on there with the multiple inheritence.
I mistyped. I meant "subclass OrderedCounter to add in the __missing__ method".
I didn't see it linked in the article nor mentioned here already but several of the problems with `super` have been discussed in "Python's Super is nifty, but you can't use it"[1] for quite some time.
[1]: http://fuhm.net/super-harmful/
[1]: http://fuhm.net/super-harmful/
That article is about Python 2's super, which is a different beast. Lessons were learnt in the design of Python 3's version.
Not really. The only thing that changed about super in Python 3 is that you can call it with no arguments, which is just a short-hand for the way it was already used in Python 2. Everything in that article applies equally to both Python versions.
Nice article, but one thing struck me when reading the code examples posted in the article: What's the use of using kwds in __init__?
class Shape(Root):
def __init__(self, **kwds):
self.shapename = kwds.pop('shapename')
super().__init__(**kwds)
class ColoredShape(Shape):
def __init__(self, **kwds):
self.color = kwds.pop('color')
super().__init__(**kwds)
class Shape(Root):
def __init__(self, shapename):
self.shapename = shapename
super().__init__()
class ColoredShape(Shape):
def __init__(self, shapename, color):
self.shapename = shapename
self.color = color
super().__init__(shapename)
The reason for using keywords is that there is no guarantee that the next class in the MRO after ColoredShape will be Shape. When the instance is a ColoredShape, it will be, but if ColoredShape is subclassed with a more complex inheritance structure, then something else could end up in between them in the MRO.
Thus, ColoredShape.__init__ can't simply pass on shapename to the super method, because that might not be the argument expected by the next __init__ method in the MRO. Instead, each method needs to gracefully accept arbitrary arguments and then pass on whatever it received (minus what it consumed).
Thus, ColoredShape.__init__ can't simply pass on shapename to the super method, because that might not be the argument expected by the next __init__ method in the MRO. Instead, each method needs to gracefully accept arbitrary arguments and then pass on whatever it received (minus what it consumed).
Am I missing some points here?
It seems that the examples are tightly coupled and favors more on inheritance than on composition.
It seems that the examples are tightly coupled and favors more on inheritance than on composition.
A true Jedi favors composition.
Good advice and well spoken :-) That being said, composing a number of object classes also involves some measure of cooperation, caller/callee ordination, and some plan for calling the methods in the correct order.
IOW, if the decomposition of the classes is the same, then the solutions using composition or using multiple inheritance will have much in common.
Things should be as simple as possible, but no simpler :-)
IOW, if the decomposition of the classes is the same, then the solutions using composition or using multiple inheritance will have much in common.
Things should be as simple as possible, but no simpler :-)
One of the reasons I'm starting to love Go more than Python.
In Go something like super() is superfluous.
In Go something like super() is superfluous.
The only reason why you don't need super in Go is because you cannot embed multiple interfaces except if their sets of methods are disjoint. This eliminates super's job of allowing programmers to differentiate between base classes with conflicting method implementations.
Without this, Go requires you to design your class hierarchy in a way that is cleaner, but the super concept is still not superfluous -- because you do sometimes need to reference an ancestor's implementation in Go.
You embed an interface and then explicitly reference the embedded interface by name. The Go interface
Without this, Go requires you to design your class hierarchy in a way that is cleaner, but the super concept is still not superfluous -- because you do sometimes need to reference an ancestor's implementation in Go.
You embed an interface and then explicitly reference the embedded interface by name. The Go interface
type ReaderWriter interface {
Reader
Writer
}
class ReaderWriter(Reader, Writer):
pass
def read_and_write(self):
super(Reader).read()
super(Writer).write()
func (rw *ReaderWriter) ReadAndWrite() {
rw.Reader.read()
rw.Writer.write()
}
Sigh. No, that is not at all how super works in Python. The equivalent code to that Go function would be:
"super(Reader).read()", if you actually try it, will just result in an AttributeError. This is because super(Reader) returns an unbound super object, not a bound super object that you can actually dispatch from. Why this advanced usage might be useful is beyond the scope of this thread, but if you're curious then just google for python unbound super and hit the Feeling Lucky button.
def read_and_write(self):
Reader.read(self)
Writer.write(self)
"super(Reader).read()", if you actually try it, will just result in an AttributeError. This is because super(Reader) returns an unbound super object, not a bound super object that you can actually dispatch from. Why this advanced usage might be useful is beyond the scope of this thread, but if you're curious then just google for python unbound super and hit the Feeling Lucky button.
Yeah you're right, I was mistaken. super uses the MRO of the first argument, or inspects the stack if none is given.
BTW your sighing is not necessary.
BTW your sighing is not necessary.
I apologize. It was late, and my supply of patience for the day was exhausted.
The "kwds" technique is actually something I came up with independently to solve a different problem when I was using PyGame: ridiculously long argument lists when constructing sprites. In this case it significantly increased the flexibility and brevity of the code, although perhaps with a more complicated inheritance tree it wouldn't be worth it.
Nice to see this pattern being acknowledged somewhere else; I thought I was the only one that did that.
Nice to see this pattern being acknowledged somewhere else; I thought I was the only one that did that.
It's a great technique. I'm sure your PyGame code benefitted in a big way :-)
[deleted]
This is a particularly unnecessary title change.
how does the runtime performance of super() compare to the hardcoding method? (from your example, using dict.__setitem__ as opposed to super().__setitem__ )
In CPython, calling super() has about the same cost as calling any other builtin like len() or int(). The MRO itself is precomputed and stored in inst.__class__.__mro__.
Contrasting dict.__setitem__ with super().__setitem__, the latter adds one function call. In addition, both forms require a builtin lookup and an attribute lookup.
In PyPy, much of the overhead of builtin lookups, function calls, and attribute lookups is automatically optimized away.
Contrasting dict.__setitem__ with super().__setitem__, the latter adds one function call. In addition, both forms require a builtin lookup and an attribute lookup.
In PyPy, much of the overhead of builtin lookups, function calls, and attribute lookups is automatically optimized away.
Unfortunately on PyPy super is less efficient than refering directly to the parent classes method, however it's fixable (and will be fixed).
I don't think it suffers much as the method resolution order (mro) is computed ahead-of-time when creating the class.
so super() is computed ahead of time? That doesn't make sense when you look at his comment:
"The calculation depends on both the class where super is called and on the instance’s tree of ancestors."
I agree that the class's MRO can be computed ahead of time, but theoretically the instance's order could be different (if a class method changed the definition of __setitem__)
"The calculation depends on both the class where super is called and on the instance’s tree of ancestors."
I agree that the class's MRO can be computed ahead of time, but theoretically the instance's order could be different (if a class method changed the definition of __setitem__)
The method resolution order for instances of the same class is always the same for all methods. If the class doesn't implement that method, the interpreter just skips and goes to the next place in the call chain. All super does is call the next method in the chain. So if you have class A(object), class B(A), class C(A), and class D(B, C), the MRO of D's instances will be B C A object. Each time an instance of D calls super, it will call the next method in the chain, not necessarily B's or C's or A's, even if the call to super is in B's method. This is why super() receives the class as a parameter, so it can know from when in the object's mro to start looking up the next method.
"This is why super() receives the class as a parameter" <-- this is not necessary in python 3. Your argument is valid in python2, but python3 allows a super() call without a type argument.
From the docs:
"super(type[, object-or-type])" <-- python2 doc, type is required
"super([type[, object-or-type]])" <-- type is optional in python3
From the docs:
"super(type[, object-or-type])" <-- python2 doc, type is required
"super([type[, object-or-type]])" <-- type is optional in python3
It's a super() good overview of super(). Nice to see a post that uses a real example.
tl;dr
Just looks like super/parent like in other languages. Did i miss something from not reading this?
Just looks like super/parent like in other languages. Did i miss something from not reading this?
> Did i miss something from not reading this?
Everything? Mostly the part where, Python being an MI language, a given class A has multiple parents. And these parents are a sequence, not a set, which define a Method Resolution Order, and that method resolution order usually isn't breadth-first either, so Python's super is a graph walker. And because it's explicit, any node in the graph can stop the walking.
This means permuting two parent classes can have significant impact on the semantics of the graph walking, therefore the results of calling super.
Everything? Mostly the part where, Python being an MI language, a given class A has multiple parents. And these parents are a sequence, not a set, which define a Method Resolution Order, and that method resolution order usually isn't breadth-first either, so Python's super is a graph walker. And because it's explicit, any node in the graph can stop the walking.
This means permuting two parent classes can have significant impact on the semantics of the graph walking, therefore the results of calling super.
Thanks for the short explanation. I totally forget about python being MI.
Some small steps in the direction of a meta-object protocol: more fine-grained control over call dispatch in a multiply-inheriting class hierarchy.
Yes, Python's super is more like Dylan's next-method than, say, C#'s super.
Yes.
Have to deal with enough languages as it is. Don't want to further add to my brain mess by learning Python 3 syntax just yet.
super() was introduced in Python2.2 many years ago. Everything in the article works with very old versions of Python.
Here's a link to the Python 2 version of the examples: http://code.activestate.com/recipes/577721-how-to-use-super-...
Here's a link to the Python 2 version of the examples: http://code.activestate.com/recipes/577721-how-to-use-super-...
super() was introduced in CPython 2.2, but it changed in Python3: http://www.python.org/dev/peps/pep-3135/. The link you posted uses super as "super(cls, instance)", but the main article uses Python 3's super, which can just be called as "super()", and it figures out the class and instance.
So no, super really has changed, and the syntax in the article does not work with very old version of Python.
So no, super really has changed, and the syntax in the article does not work with very old version of Python.
> So no, super really has changed, and the syntax in the article does not work with very old version of Python.
Oh come on, he's saying if you do the incredibly minor syntax adjustment that all of the actual meat of the article still works in Python 2.
Oh come on, he's saying if you do the incredibly minor syntax adjustment that all of the actual meat of the article still works in Python 2.
I upvoted the parent, because that syntax distinction is actually fairly giant, always having to name the class and also changing that on renames etc, is a PITA.
It is actually one of the things that annoys me most about Python, I guess this is the first thing that actually tempts me with Python 3. Damn.
It is actually one of the things that annoys me most about Python, I guess this is the first thing that actually tempts me with Python 3. Damn.
super(myclass, self) was. Like I posted below, sometimes I have real trouble keeping the 5 or so languages I use straight. The other day I actually forgot what 'None' is in python because I hadn't used it in a few days, and None is different than in the others.
Python 3 was introduced two and a half years ago. Its major syntactic changes, while frustrating if you're set in your ways, are hardly insurmountable.
This. Most of the time I run into an issue with a change I wasn't aware of, a quick search through the 3.X docs just points me in the right direction.
I think it's fine if you're doing python all day long, but I'm using Python, Java, PHP, Obj-C and Javascript each at least once a month on a variety of paid and personal projects. Like I said, I have enough trouble keeping those syntaxes straight.
I suppose similar criticism can be leveled against many other dynamic techniques, except that this one sacrifices more readability than I'm comfortable with.