More intuitive partial function application in Python(github.com)
github.com
More intuitive partial function application in Python
https://github.com/chrisgrimm/better_partial
18 comments
I'm glad you like the recipe :-)
What if you have a function "f" that takes like 10 arguments and you need to pass it to a function "g" that expects as input a function takes 8 arguments.
What if you have a function "f" that takes like 10 arguments and you need to pass it to a function "g" that expects as input a function takes 8 arguments.
@partial
def f(p1, p2, p3, p4, p5, p6, p7, p8, p9, p10):
return "stuff"
g(f(..., p4=1, p7=2))
# seems nicer than
g(lambda p1, p2, p3, p5, p6, p8, p9, p10: f(p1, p2, p3, 1, p5, p6, 2, p8, p9, p10))First mistake, making a function with 10 parameters.
We have data structures for that, if you need it. Just pass in a big struct/named tuple as one parameter. Bonus points if it is type hinted.
We have data structures for that, if you need it. Just pass in a big struct/named tuple as one parameter. Bonus points if it is type hinted.
Nit: List comprehension looks better for expressions (preferable over lambda+map)
Great to see this! Thanks for working on it.
Could you compare it to functools.partial?
In the example on the Readme:
Could you compare it to functools.partial?
In the example on the Readme:
import better_partial as bp
@bp.partial
def some_operation(x, p1, p2):
return (x + p1) * p2
func = some_operation(bp._, 10, 20)
func = lambda x: some_operation(x, 10, 20)
partial(some_operation, p1=10, p2=20)
?Yes! I definitely need to beef up the README. I wrote up a gist for some other commenters below which I can incorporate as part of the justification.
https://gist.github.com/chrisgrimm/64bca66f14528cfda6d865cc2...
https://gist.github.com/chrisgrimm/64bca66f14528cfda6d865cc2...
partial(some_operation, p1=10, p2=20)
I was gonna say cool for FP guys but alien to the Python world, but then i saw the f(..., x=1) syntax and decided you're a genius. Really nice idea.
In practice i worry about static checkers and IDE param hints though, how have you fared with these?
Not your fault at all, but mypy/pyright have very poor support for wrappers like this, and in IDEs it's common for their arg hints to devolve to a copout (args, kwargs) as well.
In practice i worry about static checkers and IDE param hints though, how have you fared with these?
Not your fault at all, but mypy/pyright have very poor support for wrappers like this, and in IDEs it's common for their arg hints to devolve to a copout (args, kwargs) as well.
> Not your fault at all, but mypy/pyright have very poor support for wrappers like this, and in IDEs it's common for their arg hints to devolve to a copout (args, kwargs) as well.
You might be able to annotate it accurately with generics, ParamSpec and Concatenate[1].
[1] https://www.python.org/dev/peps/pep-0612/
You might be able to annotate it accurately with generics, ParamSpec and Concatenate[1].
[1] https://www.python.org/dev/peps/pep-0612/
I'll give this a look.
High praise :-) I'm glad you like it! I don't know much about static checks / param hinting. TBH, I coded this up because I wanted to imagine how functions should work in Python as more of a fun exercise.
Accordingly there are a lot of things I have to do to make better_partial a strong tool for the community. I'll probably have time to work on the project again later in the week.
Accordingly there are a lot of things I have to do to make better_partial a strong tool for the community. I'll probably have time to work on the project again later in the week.
Hey, I use functools.partial a lot in my projects and I've always felt like it was pretty limited in the types of partial function application it could accomplish. I put together a modified version of partial over the weekend that I wanted to share / get feedback on. Let me know what you all think!
`functools.partial` has been bothering me for a long time and I've always thought it'd be nice to have something like the placeholder `·` that people use in math (i.e. `f(·, y, z)`). It didn't occur to me that a decorator might actually get us there without having to introduce new syntax. Nice project!
What specifically bothers you about functools partial? I think the decorator is nice in certain ways. However, then you have to decorate your code and guess where you’re going toy need the partials or create wrappers just to add the decorator which seems undesirable for a number of cases. I agree that partial has warts when it comes to inspection but at least they’re easy to make anywhere you want without having to apply any changes to the callee. Nice to have another option for partials though so that when you need these features there’s a way to get them.
I posted this below, but I can put it here too:
https://gist.github.com/chrisgrimm/64bca66f14528cfda6d865cc2...
Essentially functools.partial is somewhat limited in terms of which arguments can be partially applied.
https://gist.github.com/chrisgrimm/64bca66f14528cfda6d865cc2...
Essentially functools.partial is somewhat limited in terms of which arguments can be partially applied.
I'm glad you like it! This project was basically me trying to visualize how I wish functions generally worked like in Python.
This is a bit of a personal bugbear, but partial function application is not currying. I wrote a blog post on this a while ago because it's such a common misconception.
https://www.uncarved.com/articles/not-currying/
The link you posted is just using the word wrong. Currying is not syntactic sugar for partial application, it is the opposite. (ie currying builds functions which take multiple arguments from chains of functions which each take a single argument)
https://www.uncarved.com/articles/not-currying/
The link you posted is just using the word wrong. Currying is not syntactic sugar for partial application, it is the opposite. (ie currying builds functions which take multiple arguments from chains of functions which each take a single argument)
Yes! I almost made this mistake. Although I'm not sure @partial is the most apt name for the decorator, as it produces a function which can be partially applied. In retrospect, @partialize or something like it might be more appropriate.
@curryable looks the same as @partializable:
@curryable
def add(a, b):
return a+b
add2 = add(..., 2) # curried
print(add2(3)) # 5I think the similarity ends when you consider functions with more than 2 arguments. Let's compare a 2 argument function with a 3 argument function:
But now consider the 3 argument case:
@partial
def f3(x, y, z):
return (x, y, z)
@partial
def f2(x, y):
return (x, y)
curried_f2 = lambda x: lambda y: f2(x, y)
curried_f2(1)(2) == (1, 2)
curried_f2(1) == f2(1, _) == f2(..., x=1)
But now consider the 3 argument case:
curried_f3 = lambda x: lambda y: lambda z: f3(x, y, z)
curried_f3(1) != f3(1, _, _)
It doesn't look natural to me nesting one arg functions in Python: lambda x: lambda y: lambda z: ..
What would the harm be to adapt the abstract notion (curry) for the specific language instead of trying to apply it literally (ugly)?
For example, I remember the notion of iteration is introduced in SICP using the syntax that were it to be translated literally into Python would look like recursive calls— obviously it would be unnatural to represent iterations this way in Python.
What would the harm be to adapt the abstract notion (curry) for the specific language instead of trying to apply it literally (ugly)?
For example, I remember the notion of iteration is introduced in SICP using the syntax that were it to be translated literally into Python would look like recursive calls— obviously it would be unnatural to represent iterations this way in Python.
Can these be serialized? One of my main uses for partial is passing things to multiprocessing pool.map. People often reach for lambda but it cannot be pickled/serialized so it fails.
I'll look into this soon. I honestly wasn't expecting this much traffic on what essentially was a weekend project, so I'll need to spend a bit of time making the code less insane and testing for things like pickle-ability.
I love the idea of using underscore to represent partial currying.
Somewhat related, here's a similar PR for native support of this feature in Julia: https://github.com/JuliaLang/julia/pull/24990.
In my opinion, I'd love to see this baked in as part of the language. Getting buy in from my team to use this in production is just not going to happen. But cool package though!
Somewhat related, here's a similar PR for native support of this feature in Julia: https://github.com/JuliaLang/julia/pull/24990.
In my opinion, I'd love to see this baked in as part of the language. Getting buy in from my team to use this in production is just not going to happen. But cool package though!
Thanks! I would love to see it baked in as well. It'd be nice if, (1) all functions defaulted to behaving like better_partial.partial decorated functions and (2) similar to Ellipsis, there was some other special sentinel that we could use in place of _ (to avoid conflicting with the convention of using _ for unused variables).
I'm sure it's nice, but adding another dependency to save some keystrokes in an edge case --- no thanks. If I want lots of partial application I'll use Haskell.
Also thought the benefit too small for the drag of another dependency.
Could this be used without the "bp."? Half the point of coming up with a nicer notation is for the notation to actually be nicer.
As pointed out, you could import _ directly into your namespace, or map it to some other shorthand name that doesn't interfere with your naming conventions.
Also note that the _ placeholder isn't the only way that better_partial helps you. You can also use ... to indicate that you want to omit all arguments except ones you explicitly pass as kwargs. For instance:
Also note that the _ placeholder isn't the only way that better_partial helps you. You can also use ... to indicate that you want to omit all arguments except ones you explicitly pass as kwargs. For instance:
from better_partial import partial, _
@partial
def f(a,b,c,d,e):
return (a,b,c,d,e)
f(_,_,3,_,_)(1,2,4,5) == f(..., c=3)(1,2,4,5)Well, you could do
from better_partial import partial, _
presumably. Though since _ has some meaning in Python maybe you'd want to do
from better_partial import partial, _ as x
instead (or something like it).
from better_partial import partial, _
presumably. Though since _ has some meaning in Python maybe you'd want to do
from better_partial import partial, _ as x
instead (or something like it).
This is not a new idea. siuba (a dplyr-like Pandas alternative) is one project that does it, and it also has a "pipeline operator": https://github.com/machow/siuba
(mtcars >> group_by(_.cyl) >> summarize(avg_hp = _.hp.mean()) )
(Edit: ok, it seems siuba doesn't have the "parameter fixing" thing.)
Related: Coconut - a functional superset of Python - https://github.com/evhub/coconut
range(10) |> map$(pow$(?, 2)) |> list
There were others but I can't remember which now.
(mtcars >> group_by(_.cyl) >> summarize(avg_hp = _.hp.mean()) )
(Edit: ok, it seems siuba doesn't have the "parameter fixing" thing.)
Related: Coconut - a functional superset of Python - https://github.com/evhub/coconut
range(10) |> map$(pow$(?, 2)) |> list
There were others but I can't remember which now.
Sure, I'm not claiming to have originated the idea of using _ or some other placeholder to return partially applied functions. Lots of languages have some kind of shorthand syntax for this. I just hadn't seen it done in Python before.
Coconut looks cool btw! I'll give it a closer look soon.
Coconut looks cool btw! I'll give it a closer look soon.
Sorry, I should have worded my comment differently, I didn't mean to sound accusatory or dismissive.
No worries!
[deleted]
Interesting, but I'm wondering how does partial application for external functions works. Do you need to redefine the functions using the decorator?
One option would be to wrap the functions
from lib import external
from better_partial import partial as pp, _
pp(external)(..., arg4=10)I like how clever this is. It's a little too clever for production code for my tastes, but it might be nice to use in exploratory programming.
I really appreciate that :) Definitely give it a spin and let me know what you think!
This is great. I will use it with my python chaining approach: https://github.com/tpapastylianou/chain-ops-python
My only grief is that decorator, which forces you to wrap existing functions anyway (same way I had to define lambdas in my example anyway).
Do you have any insight on that?
My only grief is that decorator, which forces you to wrap existing functions anyway (same way I had to define lambdas in my example anyway).
Do you have any insight on that?
I'm glad you like it! I agree it's not optimal to have to wrap existing functions. One option would be to decorate the functions as they're defined, although you cant do this for external functions.
In my perfect world, python functions would default to behaving as they do when decorated with better_partial.partial, but it seems like a long-shot to try to get something like this added to the language itself.
In my perfect world, python functions would default to behaving as they do when decorated with better_partial.partial, but it seems like a long-shot to try to get something like this added to the language itself.
While it does solve the problem with functools.partial() forcing you to use keyword arguments in some cases, I don't think currying is a common enough operation to warrant a whole lib, or even a new syntax.
I'd rather see partial() promoted to a universal static method attached to any callable, so it's easier to discover, and use.
I'd rather see partial() promoted to a universal static method attached to any callable, so it's easier to discover, and use.
I guess it depends what kind of stuff you're working on. I do a lot of work with jax for my research (https://github.com/google/jax) and use functools.partial a lot.
The work you've done is nice, but using a package for this issue gives off bad code smell. I'd much rather refine my functions and their relations rather than require an extra dependency that reduces readability for my coworkers.
Thanks for the feedback! This project was really me just imagining how I'd like functions to behave in Python. I didn't really expect there to be this much interest.
> but under the hood my partial decorator is working all kinds of magic
I'm sure it is! That's why I'd be leery of using it in production code.
I'm sure it is! That's why I'd be leery of using it in production code.
:-)
whats wrong with functools.partial? I don't see that anywhere in the arguments.. all you say is "I find functools.partial unintuitive".. why? its a simple wrapper.
here's an example of one of the behaviors of functools.partial that I find unintuitive / undesirable:
https://gist.github.com/chrisgrimm/64bca66f14528cfda6d865cc2...
https://gist.github.com/chrisgrimm/64bca66f14528cfda6d865cc2...
Proposed solution:
ISTM the only time you come out a little ahead is if you know in advance that a function is going to be partialed (so you can decorate it), that you will need to partial arguments in other than a left to right fashion (otherwise a plain partial will suffice), that you won't use keyword arguments (the current partial works with keywords), and that you really dislike the lambda keyword (which neatly covers all cases from simple to complex without a new notation).
That said, this is a clever recipe. Kudos to the author for putting it together. I don't think it really rises to the level of "better" or "more intuitive", but it is an interesting experiment.