Python’s lambda is broken(math.andrej.com)
math.andrej.com
Python’s lambda is broken
http://math.andrej.com/2009/04/09/pythons-lambda-is-broken/
13 comments
Well it makes sense given the language - I just found it interesting that Haskell acts differently.
Haskell doesn't act differently. If you translate the Haskell to Lisp:
The difference is that "loop" mutates the loop variable, whereas mapping makes a copy. In Haskell, there is no way to mutate a variable (that's not even what variables are), so that's the only way you can express the concept. In Lisp, you can either copy or mutate, and you get a different result depending on which you choose.
Closures always work this way. Otherwise, you couldn't write things like:
(funcall (elt (mapcar (lambda (x) (lambda (y) (+ x y))) '(1 2 3)) 0) 2)
==> 3
The difference is that "loop" mutates the loop variable, whereas mapping makes a copy. In Haskell, there is no way to mutate a variable (that's not even what variables are), so that's the only way you can express the concept. In Lisp, you can either copy or mutate, and you get a different result depending on which you choose.
Closures always work this way. Otherwise, you couldn't write things like:
(defun make-incrementers ()
(let ((i 0))
(cons (lambda () (+ i 1))
(lambda () (- i 1)))))Similarly, if you use map instead, you get the right answer:
>>> map(lambda x: lambda i: x + i, range(0,10))[3](4)
7Perl 5.10.0 gives three different sets of answers.
Map works as the original author expected:
Map works as the original author expected:
@fs = map {sub {$_ + shift}} (0..9);
print $fs[$_](3), " " for (0..9);
>>> 3 4 5 6 7 8 9 10 11 12
push @fs, sub {$_ + shift} for (0..9);
print $fs[$_](3), " " for (0..9);
>>> 3 4 5 6 7 8 9 10 11 12
for (0..9) {push @fs, sub {$_ + shift}}
print $fs[$_](3), " " for (0..9);
>>> 3 4 5 6 7 8 9 10 11 12
for $i (0..9) {push @fs, sub {$i + shift}}
print $fs[$_](3), " " for (0..9);
>>> 3 3 3 3 3 3 3 3 3 3
for ($i=0; $i<10; $i++) {push @fs, sub {$i + shift}}
print $fs[$_](3), " " for (0..9);
>>> 13 13 13 13 13 13 13 13 13 13
I am not sure why the last two versions work differently; I'd halfway expect the bottom version, but the "for $i (0..9)" loop leaves me scratching my head.
$_ is dynamically scoped, so you are not capturing anything in your lambda. The reason they appear to work is because you are binding $_ dynamically to the right thing when you are printing the list.
Example:
Perl 5.10 lets you lexical-ize $_, but you didn't do this. Try again and let us know how it goes ;)
$_ is dynamically scoped, so you are not capturing anything in your lambda. The reason they appear to work is because you are binding $_ dynamically to the right thing when you are printing the list.
Example:
my @fs = map {sub {$_ + shift}} 0..9;
$_ = 42;
say $fs[0]->(1); # 43
Perl 5.10 lets you lexical-ize $_, but you didn't do this. Try again and let us know how it goes ;)
I understand the operational difference between using $_ and $i; it's the last two examples which confuse me, because one set of closures ends up using the very last value for $i (as I would expect for an $i scoped outside the loop), and the other uses the very first (as I would not expect at all).
None of your examples create closures. You can only close over lexicals, but all the variables you create are global. (Your loops involving $i also affect the global $i. If you want it lexically scoped, you have to say so.)
Your first two examples work because you are adding $_ and the first arg to the function. The for loop you use to print $fs[$_] binds $_, and your anonymous function uses that value. The result is something that appears to work. If you change the loop variable to something else, though, $_ will be unbound, and you will be adding to an undefined value. ("use warnings" will whine about this.)
If you then create functions that add $i and the argument, your loop stops working, because the loop binds $_ instead of $i.
Basically, you need some "my" keywords in there before your example becomes meaningful in any way. "use strict" and "use warnings" are your friend. ("use strict" will whine about the 'for $i (...)' construct that should be 'for my $i (...)', and "use warnings" will whine about $i being undefined when you try to evaluate one of the created functions.)
One last thing, if you were experimenting in Devel::REPL (re.pl), you would have noticed this problem much sooner. strict and warnings are on by default, which would have produced errors and warnings for all of your examples. It also dumps the result of evaluation with Data::Dump::Streamer, so you can actually see what's being closed over. Examples:
If we do it "right":
Your first two examples work because you are adding $_ and the first arg to the function. The for loop you use to print $fs[$_] binds $_, and your anonymous function uses that value. The result is something that appears to work. If you change the loop variable to something else, though, $_ will be unbound, and you will be adding to an undefined value. ("use warnings" will whine about this.)
If you then create functions that add $i and the argument, your loop stops working, because the loop binds $_ instead of $i.
Basically, you need some "my" keywords in there before your example becomes meaningful in any way. "use strict" and "use warnings" are your friend. ("use strict" will whine about the 'for $i (...)' construct that should be 'for my $i (...)', and "use warnings" will whine about $i being undefined when you try to evaluate one of the created functions.)
One last thing, if you were experimenting in Devel::REPL (re.pl), you would have noticed this problem much sooner. strict and warnings are on by default, which would have produced errors and warnings for all of your examples. It also dumps the result of evaluation with Data::Dump::Streamer, so you can actually see what's being closed over. Examples:
> my @fs = map { sub { $_ + $_[0] } } 1..2
$ARRAY1 = [
sub {
package Devel::REPL::Plugin::Packages::DefaultScratchpad;
use warnings;
use strict 'refs';
$_ + $_[0];
},
'V: $ARRAY1->[0]'
];
$ARRAY1->[1] = $ARRAY1->[0];
If we do it "right":
> my @fs = map { my $i = $_; sub { $i + $_[0] } } 1..2
my ($i,$i_eclipse_1);
$i = 1;
$i_eclipse_1 = 2;
$ARRAY1 = [
sub {
package Devel::REPL::Plugin::Packages::DefaultScratchpad;
use warnings;
use strict 'refs';
$i + $_[0];
},
sub {
package Devel::REPL::Plugin::Packages::DefaultScratchpad;
use warnings;
use strict 'refs';
$i_eclipse_1 + $_[0];
}
];
That was as good as perlmonks.org! You should almost have put a warning at the top, that a reader might want to think about the problem, before reading your answer. :-)
I'm convinced and will try Devel::REPL, instead of "traditional" one-liners.
I'm convinced and will try Devel::REPL, instead of "traditional" one-liners.
And not surprisingly this topic has already been discussed on Perlmonks ;-)
Here's one variation..... http://www.perlmonks.org/?node_id=719475
Here's one variation..... http://www.perlmonks.org/?node_id=719475
Ah nice. Thanks for the explanation.
Lambda is fine. The loops are mutating the variable i, so all the lambda share the same i. Apparently, list comprehensions also use mutation--I suppose that's fine.
Indeed, Python is using lexical scoping. This behavior is well known.
While true, that is not really the issue. The issue is which lexical scope the loop iterator is defined in. In python, it is apparently like this:
i = 0
increment i
<do loop body with i>
i = 0
increment 1
create new environment
bind i here
<do loop body>
The deeper issue here is "mutation is hard to reason about". Well, yeah.
Well, I would disagree - the problem is that there is no apparent mutation in the given example. There is hidden mutation, in that the list comprehension mutates its variable. This is probably done for efficiency, but I would claim it's a bad design decision.
Since that's Clojure's motto, let's see how it does:
user=> (def fs (vec (for [i (range 10)] #(+ i %))))
#'user/fs
user=> ((fs 3) 4)
7
Yes, Python closes over references to a mutable variable, and list comprehensions work the same way. It doesn't close over the value.
I'd make the suggestion that closures are not very "Pythonic". Use objects instead:
I'd make the suggestion that closures are not very "Pythonic". Use objects instead:
class add_n_func:
def __init__(self, n):
self.n = n
def __call__(self, x):
return x+self.n
return [add_n_func(n) for n in range(10)][deleted]
I would suggest that the most "Pythonic" way would be to use functools.partial and operator.add from the standard library.
As I already wrote it would look "fs = [partial(add, i) for i in range(10)]" and it works as expected.
As I already wrote it would look "fs = [partial(add, i) for i in range(10)]" and it works as expected.
How is a 5 line class with 2 underscore methods more pythonic than a nested lambda?
While python supports some basic functional programming, it's primarily a procedural/object oriented language. And it shows.
As the article demonstrates, using nested lambda's doesn't work the way you expect for closures. Using an object does.
edit: update. I found the presentation where guido talks about how he dislikes all the functional stuff, including lambda.
http://74.125.93.104/search?q=cache:2QztD1KncJgJ:www.python....
As the article demonstrates, using nested lambda's doesn't work the way you expect for closures. Using an object does.
edit: update. I found the presentation where guido talks about how he dislikes all the functional stuff, including lambda.
http://74.125.93.104/search?q=cache:2QztD1KncJgJ:www.python....
Does it make you sad that this is the same answer you'd have given if the language in question was Java?
Just rubbing it in.
Just rubbing it in.
This is rubbish. From comments right there on that site, Python and many other languages "leak" the iterated result, and combined with closures this doesn't work. The right way to do it is there:
Note for lisp knee-jerkers: I don't have anything against anonymous functions. In Python, 3rd party code is harder to understand if it has lambdas all over the place. At least for me.
HN: This is yet another example of front page trash.
EDIT:
Seems this teacher guy is not even recognizing his mistake. Now it's a "design error."
>>> fs = [(lambda n, j=i: j + n) for i in range(10)]
>>> fs[3](4)
7
Note for lisp knee-jerkers: I don't have anything against anonymous functions. In Python, 3rd party code is harder to understand if it has lambdas all over the place. At least for me.
HN: This is yet another example of front page trash.
EDIT:
Seems this teacher guy is not even recognizing his mistake. Now it's a "design error."
While I could accept the explanation about how closures capture the location
and not the value, this really is a design error. The loop variable ough to be
local to the loop. But I suppose I get what I deserve for using a dynamic language.
4.1 Naming and binding
Names refer to objects. Names are introduced by name binding operations.
Each occurrence of a name in the program text refers to the binding of that
name established in the innermost function block containing the use.
A block is a piece of Python program text that is executed as a unit.
The following are blocks: a module, a function body, and a class definition.
A script file [...] is a code block. A script command [...] is a code
block. The file read by the built-in function execfile() is a code block.
The string argument passed to the built-in function eval() and to the
exec statement is a code block. The expression read and evaluated by the
built-in function input() is a code block.Workarounds:
fs = [(lambda n, i=i: i+n) for i in range(10)]
from operator import add
from functools import partial
fs = [partial(add, i) for i in range(10)]heres what i did:
This obviously has nothing to do with the lambda in python and everything to do with the way loops work.
Python has a few quirks of this kind, this one i didn't know about. I didn't know that list comprehensions and generator expressions differ in any other way than the thing they return. I should have known better, they are very different things.
edit: looks like this also works:
>>> [i(4) for i in [(lambda n: i + n) for i in range(10)]]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <lambda>
TypeError: unsupported operand type(s) for +: 'function' and 'int'
[f(4) for f in [(lambda n: i + n) for i in range(10)]]
[13, 13, 13, 13, 13, 13, 13, 13, 13, 13]
>>> [i(4) for i in ((lambda n: i + n) for i in range(10))]
[4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
[i(4) for i in map(lambda i: lambda n:i+n, range(10))]
This obviously has nothing to do with the lambda in python and everything to do with the way loops work.
Python has a few quirks of this kind, this one i didn't know about. I didn't know that list comprehensions and generator expressions differ in any other way than the thing they return. I should have known better, they are very different things.
edit: looks like this also works:
[i(4) for i in [(lambda i: lambda n:i+n)(i) for i in range(10)]]
This issue (list comprehensions vs generator expressions) becomes clearer after reading PEP 289 (http://www.python.org/dev/peps/pep-0289/) which includes:
List comprehensions also "leak" their loop variable into the surrounding scope. This will also change in Python 3.0, so that the semantic definition of a list comprehension in Python 3.0 will be equivalent to list(<generator expression>). Python 2.4 and beyond should issue a deprecation warning if a list comprehension's loop variable has the same name as a variable used in the immediately surrounding scope.
and
After exploring many possibilities, a consensus emerged that binding issues were hard to understand and that users should be strongly encouraged to use generator expressions inside functions that consume their arguments immediately. For more complex applications, full generator definitions are always superior in terms of being obvious about scope, lifetime, and binding.
List comprehensions also "leak" their loop variable into the surrounding scope. This will also change in Python 3.0, so that the semantic definition of a list comprehension in Python 3.0 will be equivalent to list(<generator expression>). Python 2.4 and beyond should issue a deprecation warning if a list comprehension's loop variable has the same name as a variable used in the immediately surrounding scope.
and
After exploring many possibilities, a consensus emerged that binding issues were hard to understand and that users should be strongly encouraged to use generator expressions inside functions that consume their arguments immediately. For more complex applications, full generator definitions are always superior in terms of being obvious about scope, lifetime, and binding.
Thanks for the clarification. I didn't know that until i started messing with them just now for my post.
Heres a a clearer example(you could deduce this from my previous post): >>> x=((lambda n: i + n) for i in range(10)) >>> y=((lambda i: lambda n: i + n)(i) for i in range(10)) >>> [f for f in x]==[i for i in y] True
this is with generators.
with list comprehensions it becomes: >>> x=[(lambda n: i + n) for i in range(10)] >>> y=[(lambda i: lambda n: i + n)(i) for i in range(10)] >>> [f for f in x]==[i for i in y] False
Heres a a clearer example(you could deduce this from my previous post): >>> x=((lambda n: i + n) for i in range(10)) >>> y=((lambda i: lambda n: i + n)(i) for i in range(10)) >>> [f for f in x]==[i for i in y] True
this is with generators.
with list comprehensions it becomes: >>> x=[(lambda n: i + n) for i in range(10)] >>> y=[(lambda i: lambda n: i + n)(i) for i in range(10)] >>> [f for f in x]==[i for i in y] False
Ordinary Python function have the same behaviour due to Python has lexical scopes but dynamic namespaces.
http://stackoverflow.com/questions/139819/why-results-of-map...
>>> def add(n):
... return i + n
...
>>> fs = [add for i in range(10)]
>>> fs[3](4)
13
http://stackoverflow.com/questions/139819/why-results-of-map...
The difference in number of comments here (17 at the time of writing this) and on the article itself (0) shows a nice example of (lack of) usability.
I'm guessing a number of people thought about commenting on the post, then found out they had to log in, and just gave up there (at least that was my reaction). The author missed out on some interesting feedback he might've got otherwise.
[apologies for being slightly offtopic for the discussion topic, but it's related to the article]
I'm guessing a number of people thought about commenting on the post, then found out they had to log in, and just gave up there (at least that was my reaction). The author missed out on some interesting feedback he might've got otherwise.
[apologies for being slightly offtopic for the discussion topic, but it's related to the article]
Seems fine to me. If you want the captured variable to be different each time, you need to create a new environment each time. How about this?
>>> fs = [(lambda x: (lambda n: x + n))(i) for i in range(10)]
>>> [f(4) for f in fs]
[4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
>>> fs = map(lambda i: lambda n: i + n, range(10))
Same thing happens with closures in JavaScript
And your site doesn't feel that good either...
Personally I don't like this behavior (I've been bitten before in more subtle cases), but you can't blame lambda. It is consistent if one understands the same i is shared by all lambdas and loop mutates the i.