Python stands to lose its GIL, and gain a lot of speed(infoworld.com)
infoworld.com
Python stands to lose its GIL, and gain a lot of speed
https://www.infoworld.com/article/3637073/python-stands-to-lose-its-gil-and-gain-a-lot-of-speed.html
345 comments
I wonder how many folks think they're writing thread safe software with ease, and are unaware that they are leaning on the GIL?
Could be that the impact of this change is far broader than just a few key libraries.
Could be that the impact of this change is far broader than just a few key libraries.
> These changes are major enough that a fair number of existing Python libraries that work directly with Python’s internals (e.g., Cython) would need to be rewritten. But the cadence of Python’s release schedule just means such breaking changes would need to be made in a major point release instead of a minor one.
Maybe time to rethink? https://www.techrepublic.com/article/programming-languages-w...
If this is as promising as it sounds, it seems Python 4 now has its "thing" and is on the horizon. Or at least may become a serious thing to talk about
Maybe time to rethink? https://www.techrepublic.com/article/programming-languages-w...
If this is as promising as it sounds, it seems Python 4 now has its "thing" and is on the horizon. Or at least may become a serious thing to talk about
Previous discussion: https://news.ycombinator.com/item?id=28880782
The story of losing GIL is very popular in the news, and I like it too!
.. but. Let's not count our chickens until they are home. I'm wondering if the Python dev community will take on this challenge. I hope so, Sam seems to really have put in a lot of effort!
.. but. Let's not count our chickens until they are home. I'm wondering if the Python dev community will take on this challenge. I hope so, Sam seems to really have put in a lot of effort!
Design doc the proposer linked: https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsD...
In my opinion, there's a time in a language's life were it should slow down the pace of "innovation". Code bases are complexe things and updating and upgrading them constantly just to keep up with the language maybe counterproductive.
Python is there now, if you ask me. It should slow down and focus more on "maintenance" stuff with little to no impact on its interface. And maybe work on big projects like multithreading or stronger typing on the background and them when they're fully ready.
Python is there now, if you ask me. It should slow down and focus more on "maintenance" stuff with little to no impact on its interface. And maybe work on big projects like multithreading or stronger typing on the background and them when they're fully ready.
I'm looking forward to something like the brilliant rayon parallel operations library https://github.com/rayon-rs/rayon
We won't get data race free guarantees but if built into pandas or Vaex we can have a near transparent API.
It'll really open things up for those apps running on 32 core machines. They're out there, I deploy these things frequently (Plotly Dash framework for large Enterprise customers).
We won't get data race free guarantees but if built into pandas or Vaex we can have a near transparent API.
It'll really open things up for those apps running on 32 core machines. They're out there, I deploy these things frequently (Plotly Dash framework for large Enterprise customers).
These are the kinds of moves Python needs to stay relevant with amazing languages like Go out there that are just so simple yet powerful.
Benchmarks seeing a 19.4x improvement going to 20 threads, an almost linear speedup. That's pretty amazing, in Java I feel I never manage to achieve linear speedup past a few threads. How are they managing the overhead?
I use Ruby and not Python, but I think both have a lot of the same benefits and weaknesses.
IMO, removing the GIL is a major mistake. The GIL is what allows you easy concurrency and to keep the language's 'magic' while ensuring correctness. If you need parallelism, there's processes and probably other tactics (I'm not super up to date on Python things). If you simply remove the GIL you have a bunch of race conditions, so you need a bunch of new language constructs, and it just adds a bunch of complexity to solve problems that don't really need solving.
IMO they should just do what Ruby did with Ractors; basically a cheap alternative to spawning more processes. Rewriting absolutely everything that uses threads to be thread-safe is a waste of time.
IMO, removing the GIL is a major mistake. The GIL is what allows you easy concurrency and to keep the language's 'magic' while ensuring correctness. If you need parallelism, there's processes and probably other tactics (I'm not super up to date on Python things). If you simply remove the GIL you have a bunch of race conditions, so you need a bunch of new language constructs, and it just adds a bunch of complexity to solve problems that don't really need solving.
IMO they should just do what Ruby did with Ractors; basically a cheap alternative to spawning more processes. Rewriting absolutely everything that uses threads to be thread-safe is a waste of time.
> Multithreaded performance, on some benchmarks, scales almost linearly with each new thread in the best case—e.g., when using 20 threads, an 18.1× speedup on one benchmark and a 19.8× speedup on another.
Interesting. When I write Python, and I write Python most of the time, I'm not chasing performance. But when it comes to speed, a 18x speedup gets Python up to par with currently much faster languages like Java. At least if you are willing to spam threads like your life depends on it.
Interesting. When I write Python, and I write Python most of the time, I'm not chasing performance. But when it comes to speed, a 18x speedup gets Python up to par with currently much faster languages like Java. At least if you are willing to spam threads like your life depends on it.
Anyone know how far along graalpython is? I'd imagine it should be a suitable, if not superior, replacement with about as much effort right?
What is the memory model for Python after removing the GIL? Will it introduce memory barriers and atomics with different memory ordering?
I do not understand the significance of this. If i want serious numbercrunch multi-thread performance in python, i execute it on the gpu?
conda install numba & conda install cudatoolkit comes to mind?
And you should data orientated, so all that remains from objects is arrays of structures with the index being the object indicator. Im honestly puzzled about the use-case..
conda install numba & conda install cudatoolkit comes to mind?
And you should data orientated, so all that remains from objects is arrays of structures with the index being the object indicator. Im honestly puzzled about the use-case..
I don't think it will happen soon.
But what would be nice currently, is to create map reduce API w/o shared data or just readonly data.
Something like ProcessPoolExecutor but instead of spawning new process and pickling input/output data, create threads w/o GIL w/o pickling input/output data.
But what would be nice currently, is to create map reduce API w/o shared data or just readonly data.
Something like ProcessPoolExecutor but instead of spawning new process and pickling input/output data, create threads w/o GIL w/o pickling input/output data.
Knowledge of the GIL forces a lot of python developers to not try to write multithreaded scripts. Doing away with it will make life harder for many folks, IMO. Have you tried explaining multithreading to some python scripters?
I mean, it stands to evolutionary reason that Python shouldn't have a GIL.
Impressive proposal. That would remove a major limitation of CPython.
The other day:
https://news.ycombinator.com/item?id=28880782
https://news.ycombinator.com/item?id=28880782
Why is multithreaded performance important, what are the usecases where you cannot run multiple processes to spread your numbercrunching across CPUs?
I am praying for CPython to become faster. But I need faster singlethreaded performance, so web applications benefit from it.
I am praying for CPython to become faster. But I need faster singlethreaded performance, so web applications benefit from it.
pinch.take(salt)
Friend: I saw this interesting tech article ...
Me: Site?
Friend: mumble corp IT site mumble
Me: Bye
Friend: I saw this interesting tech article ...
Me: Site?
Friend: mumble corp IT site mumble
Me: Bye
I have seen and written Python code that spawns various threads with shared mutable state. Is it possible that some day the same code would run in parallel? That could be a terrible (very) breaking change. I'm not against allowing in-process parallel execution but please let it require a new API.
With all the effort that's been put into this how many people just jumped ship to a native/more-native language?
I'm probably biased because I think Python is a hacked together mess but I just don't see what the point is in dragging it around.
I'm probably biased because I think Python is a hacked together mess but I just don't see what the point is in dragging it around.
Performance doesn't come from any one quality, but from the holistic goals at each level of the language. I think some of the most frustrating aspects from the history of Python have been when the team lost focus on why and how people used the language (i.e. the 2 -> 3 transition, though I have always loved 3). I hope that this is a sensible optimization and not an over-extension.