Python idioms(jaynes.colorado.edu)
jaynes.colorado.edu
Python idioms
http://jaynes.colorado.edu/PythonIdioms.html
10 comments
The map and filter functions were almost depreciated in Python 3.
http://www.artima.com/weblogs/viewpost.jsp?thread=98196
http://www.artima.com/weblogs/viewpost.jsp?thread=98196
FTFY: depreciated --> deprecated
> I would just use
> for idx, (elt1, elt2) in enumerate(zip(list_1, list_2))
or use itertools.count:
> for idx, (elt1, elt2) in enumerate(zip(list_1, list_2))
or use itertools.count:
for idx, elt1, elt2 in zip(count(), list_1, list_2)I'm getting entirely different results with Python 3.2:
% python -m timeit "[ x**3 for x in range(10000) ]"
100 loops, best of 3: 7.85 msec per loop
% python -m timeit "map(lambda x: x**3, range(10000))"
1000000 loops, best of 3: 1.26 usec per loop
When times are that short you should question if Python is actually doing anything, or just promising to do it later (remember that range is a generator in python3).
$ python3.1 -m timeit "map(lambda x: x**3, range(10000))"
1000000 loops, best of 3: 1.01 usec per loop
$ python3.1 -m timeit "sum(map(lambda x: x**3, range(10000)))"
100 loops, best of 3: 13 msec per loop
$ python3.1 -m timeit "sum([ x**3 for x in range(10000) ])"
100 loops, best of 3: 10.6 msec per loop
$ python3.2 -m timeit "sum(map(lambda x: x3, range(10000)))"
100 loops, best of 3: 7.61 msec per loop
$ python3.2 -m timeit "sum([ x3 for x in range(10000) ])"
100 loops, best of 3: 6.27 msec per loop
$ python2.7 -m timeit "sum(map(lambda x: x3, range(10000)))"
100 loops, best of 3: 2.81 msec per loop
$ python2.7 -m timeit "sum([ x3 for x in range(10000) ])"
100 loops, best of 3: 2.28 msec per loop
I wasn't expecting python3 to be that slower.
100 loops, best of 3: 7.61 msec per loop
$ python3.2 -m timeit "sum([ x3 for x in range(10000) ])"
100 loops, best of 3: 6.27 msec per loop
$ python2.7 -m timeit "sum(map(lambda x: x3, range(10000)))"
100 loops, best of 3: 2.81 msec per loop
$ python2.7 -m timeit "sum([ x3 for x in range(10000) ])"
100 loops, best of 3: 2.28 msec per loop
I wasn't expecting python3 to be that slower.
[ldng: put some whitespace before code on hacker news: it indents it and the asterisks don't disappear.]
I infer that ldng has a 64 bit machine. On my 32 bit machine, Python 3.1 is faster than 2.6 for these examples. On a 64 bit machine I get similar results to ldng's, with Python3 being slower. If I wrap long() around the x in the example, Python2 becomes as slow as Python3.
Note that taking the cube of lots of big integers is not typical for many people: it generates very large integers that have to be in Python2's special long type on a 32 bit machine. On a 64 bit machine they stay as normal ints in Python2, which are much faster. Python3 has a single automagic int type, which seems to internally convert to the arbitrary precision type sooner than it has to on 64 bit machines(?).
Examples more typical of my use would wrap float() around the x, or change the example to add up 3x instead of x^3. These examples are all faster in Python3 for me. Faster still is to use numpy (which is now supported in Python3).
Summary: the people who would be affected by this regression have both a 64 bit machine, and do a lot of exact integer arithmetic on integers that can be represented in 64 bits, but not 32.
I infer that ldng has a 64 bit machine. On my 32 bit machine, Python 3.1 is faster than 2.6 for these examples. On a 64 bit machine I get similar results to ldng's, with Python3 being slower. If I wrap long() around the x in the example, Python2 becomes as slow as Python3.
Note that taking the cube of lots of big integers is not typical for many people: it generates very large integers that have to be in Python2's special long type on a 32 bit machine. On a 64 bit machine they stay as normal ints in Python2, which are much faster. Python3 has a single automagic int type, which seems to internally convert to the arbitrary precision type sooner than it has to on 64 bit machines(?).
Examples more typical of my use would wrap float() around the x, or change the example to add up 3x instead of x^3. These examples are all faster in Python3 for me. Faster still is to use numpy (which is now supported in Python3).
Summary: the people who would be affected by this regression have both a 64 bit machine, and do a lot of exact integer arithmetic on integers that can be represented in 64 bits, but not 32.
map() returns an iterator in Py3k.
> python3.2 -m timeit "( x**3 for x in range(10000))"
100000 loops, best of 3: 2.67 usec per loop
> python3.2 -m timeit "[ x**3 for x in range(10000)]"
100 loops, best of 3: 9.53 msec per loop
> python3.2 -m timeit "list(map(lambda x: x**3, range(10000)))"
100 loops, best of 3: 11.8 msec per loop
> python3.2 -m timeit "list( x**3 for x in range(10000))"
100 loops, best of 3: 10.7 msec per loop
> python3.2 -m timeit "list(map(lambda x: x**3, list(range(10000))))"
100 loops, best of 3: 12.1 msec per loop
> python -m timeit "map(lambda x: x**3, range(10000))"
100 loops, best of 3: 4.33 msec per loop
> python -m timeit "[x**3 for x in xrange(10000)]"
100 loops, best of 3: 2.32 msec per loop
> python -m timeit "[x**3 for x in range(10000)]"
100 loops, best of 3: 2.66 msec per loop
> pypy -m timeit "[x**3 for x in range(10000)]"
100 loops, best of 3: 2.73 msec per loop
> pypy -m timeit "[x**3 for x in xrange(10000)]"
100 loops, best of 3: 2.62 msec per loop
> pypy -m timeit "map(lambda x: x**3, xrange(10000))"
100 loops, best of 3: 2.51 msec per loop
> pypy -m timeit "map(lambda x: x**3, range(10000))"
100 loops, best of 3: 2.87 msec per loopDid you try itertools.imap instead? Vanilla map gives us a list; imap gives us a generator. (Also, I'd try a generator expression as well to see how it compares to a list comprehension).
Also, note that the article is from January 2007; I'd wager that Python has evolved and had a wealth of optimisations since then, especially regarding "common idioms" such as vanilla string concatenation and list comprehensions.
Also, note that the article is from January 2007; I'd wager that Python has evolved and had a wealth of optimisations since then, especially regarding "common idioms" such as vanilla string concatenation and list comprehensions.
> ...the article is from January 2007; I'd wager that Python has
> evolved and had a wealth of optimisations since then...
> Did you try itertools.imap instead? Vanilla map gives us a list;
> imap gives us a generator.
The rule of thumb I've read is that map is appropriate and (generally) faster when you do not need to create a lambda.
That's not strictly correct. It really comes down to whether you're calling a native python function or a C extension. If it's a native python function call, it's going to be relatively slow.
In CPython, LCs are simpler in bytecode than map/filter, which makes them faster. That's all, really.
Good stuff, though there's a few outdated idioms:
Lists, or any iterable, can be reverse-sorted with reversed(sorted(my_list)). That'll give you an iterator, though you can call list() on the result if you need it.
"while True" should be used in-place of "while 1". Reads better.
In Python 2, xrange() is preferred over range() when looping - it won't create an in-memory list of integers, and behaves mostly the same as range but for a few edge cases. Python 3 renamed xrange() to range(), and removed the original range() function.
Lists, or any iterable, can be reverse-sorted with reversed(sorted(my_list)). That'll give you an iterator, though you can call list() on the result if you need it.
"while True" should be used in-place of "while 1". Reads better.
In Python 2, xrange() is preferred over range() when looping - it won't create an in-memory list of integers, and behaves mostly the same as range but for a few edge cases. Python 3 renamed xrange() to range(), and removed the original range() function.
> reversed(sorted(my_list))
I thought sorted(my_list, reverse=True) would be slightly faster, but it seems not (by a tiny amount). Weird.
> "while True" should be used in-place of "while 1". Reads better.
If you disassemble these statements, you'll see that "while 1" creates fewer instructions:
I thought sorted(my_list, reverse=True) would be slightly faster, but it seems not (by a tiny amount). Weird.
> "while True" should be used in-place of "while 1". Reads better.
If you disassemble these statements, you'll see that "while 1" creates fewer instructions:
def foo():
while True:
pass
def bar():
while 1:
pass
import dis
dis.dis(foo)
# 0 SETUP_LOOP 10 (to 13)
# >> 3 LOAD_GLOBAL 0 (True)
# 6 POP_JUMP_IF_FALSE 12
#
# 9 JUMP_ABSOLUTE 3
# >> 12 POP_BLOCK
# >> 13 LOAD_CONST 0 (None)
# 16 RETURN_VALUE
dis.dis(bar)
# 0 SETUP_LOOP 3 (to 6)
#
# >> 3 JUMP_ABSOLUTE 3
# >> 6 LOAD_CONST 0 (None)
# 9 RETURN_VALUE
There is no difference on Python 3:
>>> dis.dis(foo)
2 0 SETUP_LOOP 3 (to 6)
3 >> 3 JUMP_ABSOLUTE 3
>> 6 LOAD_CONST 0 (None)
9 RETURN_VALUE
>>> dis.dis(bar)
2 0 SETUP_LOOP 3 (to 6)
3 >> 3 JUMP_ABSOLUTE 3
>> 6 LOAD_CONST 0 (None)
9 RETURN_VALUE~$python -mtimeit "'a' + 'b' + 'c' + 'd'"
10000000 loops, best of 3: 0.026 usec per loop
~$python -mtimeit "''.join(('a','b','c','d'))"
10000000 loops, best of 3: 0.197 usec per loop
10000000 loops, best of 3: 0.026 usec per loop
~$python -mtimeit "''.join(('a','b','c','d'))"
10000000 loops, best of 3: 0.197 usec per loop
Interesting! ''.join() has the advantage on longer strings though:
$ python -mtimeit "'aaaaaaaaaaaaaaa' + 'bbbbbbbbbbbbbbb' + 'ccccccccccccccc' + 'ddddddddddddddd'"
1000000 loops, best of 3: 0.224 usec per loop
$ python -mtimeit "''.join(('aaaaaaaaaaaaaaa','bbbbbbbbbbbbbbb','ccccccccccccccc','ddddddddddddddd'))"
10000000 loops, best of 3: 0.201 usec per loop
$ python -mtimeit "'aaaaaaaaaaaaaaa' + 'bbbbbbbbbbbbbbb' + 'ccccccccccccccc' + 'ddddddddddddddd'"
1000000 loops, best of 3: 0.224 usec per loop
$ python -mtimeit "''.join(('aaaaaaaaaaaaaaa','bbbbbbbbbbbbbbb','ccccccccccccccc','ddddddddddddddd'))"
10000000 loops, best of 3: 0.201 usec per loop
That's definitely interesting.
Always worth checking. Although the relative merits change when doing many joins to build up a single long string.
This recommendation is to go with .join() by default:
In performance sensitive parts of the library, the ''.join() form should be used instead. This will ensure that concatenation occurs in linear time across various implementations. — http://www.python.org/dev/peps/pep-0008/
despite the fact it might not be better in CPython.
This recommendation is to go with .join() by default:
In performance sensitive parts of the library, the ''.join() form should be used instead. This will ensure that concatenation occurs in linear time across various implementations. — http://www.python.org/dev/peps/pep-0008/
despite the fact it might not be better in CPython.
It's not terribly surprising. Things that scale faster tend to have higher setup costs (cf sorting algorithms; insertion sort is the fastest for relatively small n).
[deleted]
I believe the Python runtime has special-case handling for concatenating string-literals, and for concatenating string whose reference count is exactly 1. String concatenation isn't wholly defanged, though.
You're also constructing the array in the second test.
For a fair test, add elements of an array vs. joining them.
For a fair test, add elements of an array vs. joining them.
> Calling it from the empty string concatenates the pieces with no separator, which is a Python quirk and rather surprising at first.
This is stated twice, but I can't make any sense of it. How is this even remotely surprising? What else would anyone possibly expect joining with the empty string as the separator to do?
This is stated twice, but I can't make any sense of it. How is this even remotely surprising? What else would anyone possibly expect joining with the empty string as the separator to do?
I think he's referring to backwards nature of string joining in general.
Use function factories to create utility functions. Often, especially if you're using map and filter a lot, you need utility functions that convert other functions or methods to taking a single parameter. In particular, you often want to bind some data to the function once, and then apply it repeatedly to different objects. In the above example, we needed a function that multiplied a particular field of an object by 3, but what we really want is a factory that's able to return for any field name and amount a multiplier function in that family:
def multiply_by_field(fieldname, multiplier):
"""Returns function that multiplies field "fieldname" by multiplier."""
def multiplier(x):
return getattr(x, fieldname) * multiplier
return multiplier
triple = multiply_by_field('Count', 3)
quadruple = multiply_by_field('Count', 4)
halve_sum = multiply_by_field('Sum', 0.5)
There is functools.partial:
from functools import partial
def add(x,y): return x+y
add3 = partial(add, 3)
add3(2) # returns 5Ah! I didn't know about that. It'd be nice to have the syntactic sugar à la Haskell, but oh well. Close enough.
You don't need a macro to do that in a lisp. Typically you just use a built in 'partial' function, which just makes a closure that will call the original function. (It's simple, so you can just make your own if you're using a lisp without it built in.)
Use if not x instead of if x == 0 or if x == "" or if x == None or if x == False; likewise, if x instead of if x != 0, if x != None, etc.
Be careful with this one. "if not x" isn't necessarily the same as "if x == None". It's easy to forget that "if not x" will be true for values other than None.
Also, use "if x is None" rather than "if x == None". :-)
Be careful with this one. "if not x" isn't necessarily the same as "if x == None". It's easy to forget that "if not x" will be true for values other than None.
Also, use "if x is None" rather than "if x == None". :-)
I read many years ago that '%s%s' % (a,b) was faster than a + b, the reason given that it was done in C. But after reading this thread and trying myself, it seems to be false also. On Py 2.6:
python -m timeit " '%s%s%s' % ('aaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbbbbbbbb', 'ccccccccccccccccccccccccccccccc') "
1000000 loops, best of 3: 0.201 usec per loop
python -m timeit " 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaa' + 'bbbbbbbbbbbbbbbbbbbbbbbb' + 'ccccccccccccccccccccccccccccccc' "
10000000 loops, best of 3: 0.136 usec per loop
Use whatever is clearest to read, since the benchmarks vary upon circumstance, VM, and VM version.
For what is worth, most "performance" idioms are anti-idioms on PyPy. Especially those:
sum = 0
for d in data:
sum += d
product = 1
for d in data:
product *= d
What's the rationale for preferring map/filter over list comprehensions?
I wish every language listed it's idioms somewhere.
I would just use
The following is not necessarily good advice, and he should take his own earlier advice and validate his claims with some profiling.
Note that the following "map" snippet is considerably slower than the corresponding list-comprehension snippet. Function calls are expensive in python. Also, his claim that you save time with a "map" by avoiding the loop variable is obviously bogus: you still have to bind the variables in the signature of the function you're mapping, unless the signature is empty.