PyCon 2015: Are we still changing the world?(blog.lisnr.com)
blog.lisnr.com
PyCon 2015: Are we still changing the world?
http://blog.lisnr.com/2015/05/11/pycon-2015-are-we-still-changing-the-world/
11 comments
Our pragmatism is one of the biggest factors that draws me to Python as a dev.
I agree about our significant use outside of web dev -- even though I know a lot of it is "out there", I had a hard time gathering it with many major orgs not being on CrunchBase, AngelList, or StackShare, (and some with a footprint that doesn't represent their true size on GitHub). If there's a source for this data out there, I'd love to find it.
I agree about our significant use outside of web dev -- even though I know a lot of it is "out there", I had a hard time gathering it with many major orgs not being on CrunchBase, AngelList, or StackShare, (and some with a footprint that doesn't represent their true size on GitHub). If there's a source for this data out there, I'd love to find it.
Yes, I can confirm that I've done a prodigious amount of system scripting in Python.
Is Python changing the world, or Go, or Ruby, or Javascript? If you are asking about "changing the world" that's a big thing to ask for. Simply being useful doesn't make a technology world-changing. I feel very confident that Go will not change the world. People may use Go to make things that change the world, but Go won't change the world. If Python has changed the world – that is the language itself has done so, not just the people using the language – then I'd look mostly to its community, governance, and development process. For Ruby I'd look to its culture, philosophy, and discourse. Javascript actually has a technical claim to changing the world: it's created a programming platform that requires no installation of software, and has brought sandboxed development into the world in a broad and practical way. But if we look at Node.js/IO.js, a Javascript platform with similar scope to Python, then I personally see a generational reinvention: a platform that a new generation of budding developers can call their own, and make their own decisions. Asking a lot of questions that no one actively considers in these older platforms.
But there's lots of other things that can change the world besides programming languages. Are programming languages really the best place to hang your hat? Do you want to compare Python to Ruby or Node.js, or compare Python a web application or wiki or the cloud; i.e., entirely different categories?
But there's lots of other things that can change the world besides programming languages. Are programming languages really the best place to hang your hat? Do you want to compare Python to Ruby or Node.js, or compare Python a web application or wiki or the cloud; i.e., entirely different categories?
Ian -- I'm honored for you taking the time to read through and comment. Your post (understandably) has the highest click-through of anything I linked. At first I was a bit taken back by the words myself (borrowed from the PSF [0]), but as you've pointed out, all of these languages have demonstrably changed the world through different aspects. Thank you for your insight and efforts in sparking the conversation.
0: http://pyfound.blogspot.com/2015/02/psf-python-brochure-sold...
0: http://pyfound.blogspot.com/2015/02/psf-python-brochure-sold...
"People may use $LANGUAGE to make things that change the world, but $LANGUAGE won't change the world."
Which makes total sense.
It's the things (end product) that matter, and unless they could only/better/easier be implemented in $LANGUAGE, then it could have been many other languages that could have been used as well.
Whereas something like C did change the world.
It's the things (end product) that matter, and unless they could only/better/easier be implemented in $LANGUAGE, then it could have been many other languages that could have been used as well.
Whereas something like C did change the world.
Right now, a huge opportunity for Python seems to be data science. I didn't notice it in the list of places where Python shows up. Also, in that space, it more than just shows up, it brings unique capability to the space with its libraries and the support major tool developers have for it.
I think folks just don't talk about Python, they get things done.
While I think, say, other areas also get things done, Rails developers are more likely to blog and have pretty fonts. Nothing against that, it just seems to be the case.
While metrics show there's more Python out there, it's just not exciting, it doesn't change a lot, and it just works. And that's fine too.
While I think, say, other areas also get things done, Rails developers are more likely to blog and have pretty fonts. Nothing against that, it just seems to be the case.
While metrics show there's more Python out there, it's just not exciting, it doesn't change a lot, and it just works. And that's fine too.
I find myself thinking (hoping) this is the case too. In fact, writing the post to talk about Python at this level was indirectly inspired by a talk I heard at Code Genius (http://code.genius.com/), put on by the Ruby-based Rap Genius Engineering team.
With respect to metrics, I think having the baseline is important, and you're right. I also found myself looking for more granular metrics then what I was able to find. I really want to be able to answer questions like
- "Within Python, how is usage shifting between web apps, scientific work (both research prototyping and in practice), open source, machine learning, etc.?" - "How are the 'frameworks on top of frameworks' like Flask-RESTful and Django Rest Framework changing in adoption over time?" - "Is anyone actually adopting Python 3 as their default yet?"
I wasn't quite able to find that data, but if anyone has ideas for first steps in getting there, I'd be happy to contribute (email in profile).
With respect to metrics, I think having the baseline is important, and you're right. I also found myself looking for more granular metrics then what I was able to find. I really want to be able to answer questions like
- "Within Python, how is usage shifting between web apps, scientific work (both research prototyping and in practice), open source, machine learning, etc.?" - "How are the 'frameworks on top of frameworks' like Flask-RESTful and Django Rest Framework changing in adoption over time?" - "Is anyone actually adopting Python 3 as their default yet?"
I wasn't quite able to find that data, but if anyone has ideas for first steps in getting there, I'd be happy to contribute (email in profile).
On my end (east coast US), I hear the Python/data/analytics things a lot but haven't directly encountered those companies.
Python was at least previously big in a lot of local shops doing a lot of systems programming type stuff (management apps) -- including folks like Red Hat and HP, including my old one (Ansible). Startups are still highly likely to use it.
There are enough Django shops around, including at least one major Django consultancy-type shop, but it feels like slightly more (but not much more) rails on the web side.
I'm not plugged into the industry biotech to know how much Python is floating around, but I think it's probably pretty common, just that those folks travel in slightly different circles and we don't cross polinate enough.
"Is anyone actually adopting Python 3 as their default yet?"
Almost certaintly someone has, the real question is what percentage and what characterizes them :)
Python was at least previously big in a lot of local shops doing a lot of systems programming type stuff (management apps) -- including folks like Red Hat and HP, including my old one (Ansible). Startups are still highly likely to use it.
There are enough Django shops around, including at least one major Django consultancy-type shop, but it feels like slightly more (but not much more) rails on the web side.
I'm not plugged into the industry biotech to know how much Python is floating around, but I think it's probably pretty common, just that those folks travel in slightly different circles and we don't cross polinate enough.
"Is anyone actually adopting Python 3 as their default yet?"
Almost certaintly someone has, the real question is what percentage and what characterizes them :)
Python is still over shadowed by R in data science.
I stopped using Python for data science and use R and it really worked out great for me. R has gained a lot of momentum and I think that Pandas with Wes leaving isn't being developed as quickly. (Most of it is re-doing what R has done). I am amazed at all the FUD I was told to not use R and just use Python but after hitting so many walls I tired it and I am so glad I did. Your millage may vary.
I stopped using Python for data science and use R and it really worked out great for me. R has gained a lot of momentum and I think that Pandas with Wes leaving isn't being developed as quickly. (Most of it is re-doing what R has done). I am amazed at all the FUD I was told to not use R and just use Python but after hitting so many walls I tired it and I am so glad I did. Your millage may vary.
R is an awful language with great libraries.
My main beef about R (and similar tools) is that I summarize my data I get from Hadoop and find it is still too big for R to handle and I then have to summarize it some more.
My main beef about R (and similar tools) is that I summarize my data I get from Hadoop and find it is still too big for R to handle and I then have to summarize it some more.
R does not deserve to be called awful. It is a language that works for many people, people make a living from it and is one of the fastest growing languages.
[1] https://blog.javascripting.com/2014/07/28/fastest-growing-ne...
[2] http://www.pcworld.com/article/2480920/r-programming-languag...
[3] http://redmonk.com/sogrady/2015/01/14/language-rankings-1-15...
Yes I have read [R Inferno] http://www.burns-stat.com/pages/Tutor/R_inferno.pdf
[1] https://blog.javascripting.com/2014/07/28/fastest-growing-ne...
[2] http://www.pcworld.com/article/2480920/r-programming-languag...
[3] http://redmonk.com/sogrady/2015/01/14/language-rankings-1-15...
Yes I have read [R Inferno] http://www.burns-stat.com/pages/Tutor/R_inferno.pdf
Have you tried the Hadley Universe? Seriously there isn't a data set that Python can do that R can't (Lots of ways to do things as opposed to Python's one correct way). Using dplyr has been a huge change in the way I do any data munging. Panda's has started using some of dplyr with new function assign. Dplyr really handles large data sets well like data.tables (data.table is fast and handles larger sets but I like dplyr's syntax more)
By the way i should have stated that Python is a good choice for doing Data Science it is just that R works better for me.
By the way i should have stated that Python is a good choice for doing Data Science it is just that R works better for me.
As others have said, try data.table.
You assessment is pretty accurate. R is almost the opposite of Python -- you HAVE to use lots of external libraries. Whereas in Python, the core is super solid, but the libraries you get are of varying quality.
R took me the longest to learn out of any language. I currently have coworkers struggling with R, and cursing it. But the learning pays off... it's pretty much the only option out there right now (and yes I use Python for almost everything, but you need R too.)
You assessment is pretty accurate. R is almost the opposite of Python -- you HAVE to use lots of external libraries. Whereas in Python, the core is super solid, but the libraries you get are of varying quality.
R took me the longest to learn out of any language. I currently have coworkers struggling with R, and cursing it. But the learning pays off... it's pretty much the only option out there right now (and yes I use Python for almost everything, but you need R too.)
Have you tried data.table?
Thanks for your insight. I have some questions for your, if you don't mind.
What walls and why did you switch? Did you find the shift in syntax and mindset difficult? Have you used rpy2?
There definitely seems to be more innovation in R world, but blaze, numba and bokeh have no R equal.
What walls and why did you switch? Did you find the shift in syntax and mindset difficult? Have you used rpy2?
There definitely seems to be more innovation in R world, but blaze, numba and bokeh have no R equal.
Walls:
I work at a Microsoft Office Universe (They refuse to know of any tools besides Word and you are a tech god if you use Excel) So when I finish my reports in R I actually give them Office Documents. The presentation is actually Power Point and the documents are in Word.
I left Python and Pandas and the visualization was far behind what R had in its tool chain. I still feel that R does visualizations best.
RStudio (A true IDE) was better for my work flow than IPython/Juypter. I actually really missed IPython at first and than slowly I grew to love RStudio.
Not a wall, maybe a paper cut. A 1 index (R) over a 0 index (Python) is pretty smart for doing statistics, less errors.
I also liked the flexibility to use whatever I wanted to do my work. Was difficult at first than learned to love the fact that there were 3 good ways to do something.
Functional Programming. You can do some good functional programming with R. Functions are first class citizens.
Biggest reason for anyone to switch is Hadley Wickham and his work with RStudio. It really is amazing everything that he has done for R in the last 5 years.
Once again Python will get the job done but these are some of the reasons why I switched.
I work at a Microsoft Office Universe (They refuse to know of any tools besides Word and you are a tech god if you use Excel) So when I finish my reports in R I actually give them Office Documents. The presentation is actually Power Point and the documents are in Word.
I left Python and Pandas and the visualization was far behind what R had in its tool chain. I still feel that R does visualizations best.
RStudio (A true IDE) was better for my work flow than IPython/Juypter. I actually really missed IPython at first and than slowly I grew to love RStudio.
Not a wall, maybe a paper cut. A 1 index (R) over a 0 index (Python) is pretty smart for doing statistics, less errors.
I also liked the flexibility to use whatever I wanted to do my work. Was difficult at first than learned to love the fact that there were 3 good ways to do something.
Functional Programming. You can do some good functional programming with R. Functions are first class citizens.
Biggest reason for anyone to switch is Hadley Wickham and his work with RStudio. It really is amazing everything that he has done for R in the last 5 years.
Once again Python will get the job done but these are some of the reasons why I switched.
Interesting thanks for sharing. Hopefully bokeh might be able to address the visualization space with its upcoming statistical glyphs.
When you say "flexibility to use whatever I wanted to do my work", do you mean data munging and manipulation or statistical modeling packages?
When you say "flexibility to use whatever I wanted to do my work", do you mean data munging and manipulation or statistical modeling packages?
Programming I mean that I have 5 different ways of doing anything and everything. SO if I want to do loops (I never do and people shouldn't) I can. If I want to do strict funtional programming, no problem. If I want to use a SQL type syntax I can use data.table. This is frustrating at first since stackoverflow will have a bunch of solutions and it will be something new. I end up really using a few libraries and sticking with them. I use dplyr, stringr, httr, ggvis or ggplot2 and tidyr.
Graphics I can use just about anything. I can use plottly, ggplot2, D3.js, PDF report, HTML, send to a blogging site, interactive graphics with ggvisand Shiney, even PowerPoint slides.
Graphics I can use just about anything. I can use plottly, ggplot2, D3.js, PDF report, HTML, send to a blogging site, interactive graphics with ggvisand Shiney, even PowerPoint slides.
Functions are not first-class citizens in Python?
<-- Maturity -- Fundamentals well designed -->
R Python JuliaThe relationship between maturity and design with respect to usefulness is not linear, with respect to "usefulness in production".
Right, but it does depend on how long term you view it. Problems in the fundamentals of the core language can't be fixed, a mature library ecosystem can be developed.
I also have the same feeling. Python is very mathematics-friendly compared to other imperative languages and is used a lot more than other mainstream languages for mathematics related projects. On Github, there is a lot of good quality machine learning and statistics projects built with Python.
I agree with you 100%. I also got a comment from a NASA engineer that other strong additions are NASA itself and the Department of Energy. If you know of any specific companies, I'd definitely welcome the suggestion.
For me, the only place where Python truly excels versus the competition is Numpy. The easy python scaffolding gets you to the heart of a problem in a friendly and flexible way, and then you can interface easily to this big V8 of a compute engine (especially if you combine it with the MKL libraries). The ecosystem around Numpy is astronomically good.
I have fallen in love again with Python (2) for this reason. I tried 3 but I don't need all it's accountant-like seriousness. It's less fun to code in. When I need to design for servers or for performance I will do golang or ocaml. When I need simple scaffolding for numerical work, and scripting for prototyping integration of various systems, Python 2.7 is like an old, unfussy, friend, helping me along my long and winding data wrangling road with a minimum of cognitive tax. It just works. Long may it live.
In my view the Python powers should look at numerical / data science and throw all of Python's weight behind that, and forget about the web, where they've already lost.
I have fallen in love again with Python (2) for this reason. I tried 3 but I don't need all it's accountant-like seriousness. It's less fun to code in. When I need to design for servers or for performance I will do golang or ocaml. When I need simple scaffolding for numerical work, and scripting for prototyping integration of various systems, Python 2.7 is like an old, unfussy, friend, helping me along my long and winding data wrangling road with a minimum of cognitive tax. It just works. Long may it live.
In my view the Python powers should look at numerical / data science and throw all of Python's weight behind that, and forget about the web, where they've already lost.
Can you articulate what you mean by Python 3's "accountant-like seriousness"? Is there more to it than print-as-a-function?
I code in both every day and just don't see what the fuss is about. To me, they're barely different dialects; more like different accents.
I code in both every day and just don't see what the fuss is about. To me, they're barely different dialects; more like different accents.
First of all I don't think the print statement is a small issue. I bet 5% of my code is print statements. I love the print statement's unfussy "can do" mentality. Hey, the whole reason I got into Python was because it was utterly barebones. Print "hello world". that's it. You can almost feel the fun a young GvM might have had making it.
Then all this u"xx" stuff on strings. What's that about? I don't care about unicode. 256 ascii characters is fine for me. If I need Unicode I can do it, but I don't need it by default.
Iterators instead of ranges. Fine. Why again? To save memory? I've got 32 billion bytes. I don't need it. It's complicating something to please the computer scientists and it's messing with my mind which has far bigger problems to solve, so it's unpragmatic. If I see htop moaning about memory, I can easily change my code.
Generally, I just like the 2.0 attitude. It's carefree. It just works. Three is trying too hard. Python is just a tool for me. My really hard problems are my daily battles with nonconvex optimization of vast data sets.
Then all this u"xx" stuff on strings. What's that about? I don't care about unicode. 256 ascii characters is fine for me. If I need Unicode I can do it, but I don't need it by default.
Iterators instead of ranges. Fine. Why again? To save memory? I've got 32 billion bytes. I don't need it. It's complicating something to please the computer scientists and it's messing with my mind which has far bigger problems to solve, so it's unpragmatic. If I see htop moaning about memory, I can easily change my code.
Generally, I just like the 2.0 attitude. It's carefree. It just works. Three is trying too hard. Python is just a tool for me. My really hard problems are my daily battles with nonconvex optimization of vast data sets.
> Then all this u"xx" stuff on strings. What's that about? I don't care about unicode. 256 ascii characters is fine for me. If I need Unicode I can do it, but I don't need it by default.
I felt this way before I started spending all of my time on web apps. It's reading user input data from some random public source, like Twitter, that forces it upon you. Then, so quickly it became the best practice to "unicode all the things". I think of analogous to how we store timestamps in UTC always.
I felt this way before I started spending all of my time on web apps. It's reading user input data from some random public source, like Twitter, that forces it upon you. Then, so quickly it became the best practice to "unicode all the things". I think of analogous to how we store timestamps in UTC always.
One difference, though, is that time enjoys a certain natural and intrinsic consensus. For example, we all agree that observable time always flows forward at the same rate.
OTOH: Which characters do and don't belong in unicode and in what order? I don't fucking know. :-)
OTOH: Which characters do and don't belong in unicode and in what order? I don't fucking know. :-)
> OTOH: Which characters do and don't belong in unicode and in what order? I don't fucking know. :-)
Should we use decimalized time or time based on the Babylonian base 60/12 system? Both have clear advantages. I don't fucking know. :-)
The world has standardized on Unicode, which (as a collection of expanding standards) defines the set of valid code points and their order. There's still some debate as to UTF-8 vs. UTF-16LE (and perhaps UTF-16 w/BOM and UTF-32) encodings, but Unicode has clearly won. It's not perfect, but it's silly to pretend Unicode hasn't won.
Source: I used to work as an engineer on the content converter portion of Google's indexing system, which took the world's web pages, PDFs, etc. and converted them into a unified format (the text portion of which is encoded as UTF-8) for the rest of the indexing system. Sure, we saw some percentage of EUC-KR, GB2312, Big5, and Win CP1252 text, but Unicode has clearly won and UTF-8 and UTF-16LE are steadily replacing all other encodings.
Should we use decimalized time or time based on the Babylonian base 60/12 system? Both have clear advantages. I don't fucking know. :-)
The world has standardized on Unicode, which (as a collection of expanding standards) defines the set of valid code points and their order. There's still some debate as to UTF-8 vs. UTF-16LE (and perhaps UTF-16 w/BOM and UTF-32) encodings, but Unicode has clearly won. It's not perfect, but it's silly to pretend Unicode hasn't won.
Source: I used to work as an engineer on the content converter portion of Google's indexing system, which took the world's web pages, PDFs, etc. and converted them into a unified format (the text portion of which is encoded as UTF-8) for the rest of the indexing system. Sure, we saw some percentage of EUC-KR, GB2312, Big5, and Win CP1252 text, but Unicode has clearly won and UTF-8 and UTF-16LE are steadily replacing all other encodings.
I think you're confused about the strings. Python 3 uses UTF8 strings by default. The u"xx" syntax is only needed in python 2.x. In Python 3.x it's only supported for backwards compatibility.
right thanks for clarifying - I wasn't confused until Python 3 made it confusing.
EDIT: Ah I remember. In one of my attempts to move to 3 I wrote a few modules which worked with strings, then I had to stick "u"s everywhere when I passed them to my 2 code base.
> Then all this u"xx" stuff on strings. What's that about? I don't care about unicode. 256 ascii characters is fine for me. If I need Unicode I can do it, but I don't need it by default.
But plenty of people were writing code that conflated bytes and strings and broke in subtle ways on systems with non-English locales. Maybe you don't notice the bugs, so you don't care, but that doesn't mean that it's not a major flaw in Python 2. Here in Asia, plenty of people may be cursing your silently buggy Python 2 unicode-naive code.
It's similar to a pair of time-handling functions in a domain-specific language I use at work sometimes. The lazy way of converting Times to Strings (arguably reasonably) uses the current processes's time zone to render the Time. Likewise, Time's constructor uses the current processes's time zone when constructing a Time form a String. Someone decided to write a pair of functions that shift a given time forward and backward by the current processes's time zone's UTC offset on the day in question. The one function gives you a Time that if rendered using lazy String formatting (usually) gives you the same String you would have gotten if you had used a function that took a time zone as an argument and passed UTC as the time zone. The inverse function allows you to take a String representation of a UTC time, pass it to Time's constructor without specifying a time zone, and then shift that time so that you (usually) get the same Time you would have gotten had you specified the correct time zone in Time's constructor. This sounds absolutely insane, but it mostly works, except in corner cases that cross DST changes. (Note that 2014-03-09 02:31 America/New_York just doesn't exist, but 2014-03-09 02:31 UTC is a perfectly sensible time.) We now have a jslint-like program that issues a warning if you use either of these functions. Freely guessing at the correct conversion back and forth from byte arrays to code points is very much like this insanity of mostly-working functions to shift times to make their conversions to and from Strings mostly work without having to specify time zones in the Time or String constructors.
Similarly, if your language silently converted April 74th 2014 to June 3rd 2014, it would "just work" for 99.9% of use cases, but it would also hide some bugs. People would complain if the language's designers changed the language's next major version to stop accepting such non-sensical dates. Silently converting back and forth between bytes and code points is more subtle, but similarly insane in hiding bugs that 99.9% of the time would go unnoticed.
Sure, silently ignoring locale conversions of times and strings mostly just works, but that doesn't mean it's not terribly broken. PHP rightly gets a lot of criticism for often following the philosophy of "do what I probably meant, and it will just work 99.9% of the time and cause insane bugs 0.1% of the time".
But plenty of people were writing code that conflated bytes and strings and broke in subtle ways on systems with non-English locales. Maybe you don't notice the bugs, so you don't care, but that doesn't mean that it's not a major flaw in Python 2. Here in Asia, plenty of people may be cursing your silently buggy Python 2 unicode-naive code.
It's similar to a pair of time-handling functions in a domain-specific language I use at work sometimes. The lazy way of converting Times to Strings (arguably reasonably) uses the current processes's time zone to render the Time. Likewise, Time's constructor uses the current processes's time zone when constructing a Time form a String. Someone decided to write a pair of functions that shift a given time forward and backward by the current processes's time zone's UTC offset on the day in question. The one function gives you a Time that if rendered using lazy String formatting (usually) gives you the same String you would have gotten if you had used a function that took a time zone as an argument and passed UTC as the time zone. The inverse function allows you to take a String representation of a UTC time, pass it to Time's constructor without specifying a time zone, and then shift that time so that you (usually) get the same Time you would have gotten had you specified the correct time zone in Time's constructor. This sounds absolutely insane, but it mostly works, except in corner cases that cross DST changes. (Note that 2014-03-09 02:31 America/New_York just doesn't exist, but 2014-03-09 02:31 UTC is a perfectly sensible time.) We now have a jslint-like program that issues a warning if you use either of these functions. Freely guessing at the correct conversion back and forth from byte arrays to code points is very much like this insanity of mostly-working functions to shift times to make their conversions to and from Strings mostly work without having to specify time zones in the Time or String constructors.
Similarly, if your language silently converted April 74th 2014 to June 3rd 2014, it would "just work" for 99.9% of use cases, but it would also hide some bugs. People would complain if the language's designers changed the language's next major version to stop accepting such non-sensical dates. Silently converting back and forth between bytes and code points is more subtle, but similarly insane in hiding bugs that 99.9% of the time would go unnoticed.
Sure, silently ignoring locale conversions of times and strings mostly just works, but that doesn't mean it's not terribly broken. PHP rightly gets a lot of criticism for often following the philosophy of "do what I probably meant, and it will just work 99.9% of the time and cause insane bugs 0.1% of the time".
The many more generator-style iterables give it a strange dramatic tone to me. Also, the type hints syntax looks utterly un-fun.
You can ignore the type hints syntax. I plan to.
Python 2:
Python 2:
map(func, iterable) # returns a list
map(func, iterable) # returns an iterator
list(map(func, iterable)) # returns a list> You can ignore the type hints syntax. I plan to.
I feel like that will be quite an ear-plugging, eye-covering endeavor.
And that's really the point: it has the potential to create a community in which half the people are eyes-closed, ears-plugged.
On your second point: Yeah, I don't mind casting things to a list. But it does seem... I guess, dramatic.
I feel like that will be quite an ear-plugging, eye-covering endeavor.
And that's really the point: it has the potential to create a community in which half the people are eyes-closed, ears-plugged.
On your second point: Yeah, I don't mind casting things to a list. But it does seem... I guess, dramatic.
> And that's really the point: it has the potential to create a community in which half the people are eyes-closed, ears-plugged.
This is a big concern of mine as well, both before/after listening to Guido's keynote on Type Hinting at PyCon (https://www.youtube.com/watch?v=2wDvzy6Hgxg).
At first I thought type hinting would only really see adoption in performance-oriented sections of code, but I think we're all curious to see what best practices emerge in general, and in the open source side of things. Really the additional robustness it unlocks for IDEs like JetBrains is what got me really excited about type hints in Python.
This is a big concern of mine as well, both before/after listening to Guido's keynote on Type Hinting at PyCon (https://www.youtube.com/watch?v=2wDvzy6Hgxg).
At first I thought type hinting would only really see adoption in performance-oriented sections of code, but I think we're all curious to see what best practices emerge in general, and in the open source side of things. Really the additional robustness it unlocks for IDEs like JetBrains is what got me really excited about type hints in Python.
> Really the additional robustness it unlocks for IDEs like JetBrains is what got me really excited about type hints in Python.
Agreed. But that's only useful if the syntax is non-awful.
Agreed. But that's only useful if the syntax is non-awful.
It's not technically a cast to a list. It's using the list constructor with the iterator as an argument.
Thanks for taking the time to read. I agree with most of what you've said, but of course I still don't think Python has lost on the web, though that was my concern that sparked writing this post.
What are your thoughts on the future of Python + NumPy vs. MATLAB or R? It seems the numeric and scientific community is embracing Python more, but what I've heard is mostly anecdotal.
What are your thoughts on the future of Python + NumPy vs. MATLAB or R? It seems the numeric and scientific community is embracing Python more, but what I've heard is mostly anecdotal.
I use R extensively, even more than Python, but it is not a robust language when your problem moves from exploration to anything resembling production. The memory management appears to be the problem. Hugely inefficient and slow even with 3.x. I get crashes frequently, even under Linux and I run production on R. Moreover, Numpy's two orders-of-magnitude speed increase opens up a new class of problem that R just can't handle, especially in the streaming data space. Pandas has made a big improvement in taking Python towards R, though it would be great if it were multidimensional. Where R has Python beat hands down is in its "vectors everywhere" functional philosophy. List comprehensions just don't match the power, and vector indexing is uneven even in Pandas.
I have less experience with Matlab for cost, and lock in reasons. I need to have the option to distribute my code easily.
I have less experience with Matlab for cost, and lock in reasons. I need to have the option to distribute my code easily.
toolz is pretty good at lazy fp in python : https://github.com/pytoolz/toolz
Is the Python 2 vs Python 3 thing ever going to be resolved?
If you look at Python adoption from the perspective of someone picking it up for the first time, the very first issue they encounter is the "which version of Python should I use" debate, which pretty much frames the experience from that point onward.
The pragmatic attitude that's kept Python 2.X afloat is a key part of why Python has had longevity but it also doesn't set a great tone for "changing the world" or keeping up with the latest sexy fads
If you look at Python adoption from the perspective of someone picking it up for the first time, the very first issue they encounter is the "which version of Python should I use" debate, which pretty much frames the experience from that point onward.
The pragmatic attitude that's kept Python 2.X afloat is a key part of why Python has had longevity but it also doesn't set a great tone for "changing the world" or keeping up with the latest sexy fads
Ubuntu 16.04 LTS is going to be Python 3 by default. I switched to Python 3 only a year ago. The war is over, Python 3 won, and now it's just the "too big to fail" guys who still think Python 2 has a future. Google, Dropbox and a few others who will probably give it another5 years before they make the switch using 3.5's type hints to help automate the process of moving to Python 3.
There was never a war, and Python 3 didn't win it.
We're still not seeing any mainstream Python 3 adoption (no, I don't count your own project). PyPI downloads was still < 5% last I checked. The big web frameworks are still better tested and better performant with 2. Scientific computing is stuck with 2. Same with Ansible. And the big companies you mention have no (public) intention to migrate.
The core developers decided long ago that 3 is what everyone should use, so I have no doubt that will be the case. The open question is how much is left of the ecosystem by then. It's commonly discussed at conferences but there is no straightforward answer. At least we got the TLS stuff backported to 2 which might be indicative of a coming thawing.
We're still not seeing any mainstream Python 3 adoption (no, I don't count your own project). PyPI downloads was still < 5% last I checked. The big web frameworks are still better tested and better performant with 2. Scientific computing is stuck with 2. Same with Ansible. And the big companies you mention have no (public) intention to migrate.
The core developers decided long ago that 3 is what everyone should use, so I have no doubt that will be the case. The open question is how much is left of the ecosystem by then. It's commonly discussed at conferences but there is no straightforward answer. At least we got the TLS stuff backported to 2 which might be indicative of a coming thawing.
I don't understand how python's leadership works but from my outsider perspective it seems like the solution is that those with some power need to show some muscle in forcing a direction. I mean this has been going on for 7 years now, someone needs to lay down the law.
If you introduce a new developer to Python the first step is "ignore Python 3, nobody uses that" and then explain that there's infighting about backwards incompatibility so most packages either don't support it or do it badly. Why do I have to justify to new talent the reasons why I'm using a language that can't even agree on the latest version?
If you introduce a new developer to Python the first step is "ignore Python 3, nobody uses that" and then explain that there's infighting about backwards incompatibility so most packages either don't support it or do it badly. Why do I have to justify to new talent the reasons why I'm using a language that can't even agree on the latest version?
> We're still not seeing any mainstream Python 3 adoption
I think mainstream adoption is increasing, but not as quickly as anyone would like. The sentiment around adoption I felt echoed at PyCon this year was: "we have dependencies that aren't ported to 3 yet [... and who will be the ones to port them?!]".
I think mainstream adoption is increasing, but not as quickly as anyone would like. The sentiment around adoption I felt echoed at PyCon this year was: "we have dependencies that aren't ported to 3 yet [... and who will be the ones to port them?!]".
The Python 2 codebase is growing much faster than Python 3, because the Python core developers have lost touch with the fact that its most dedicated users are not web people, but scientific computing users for who the overwhelming choice is 2.7. Anaconda gives you a 3.x choice on its website almost as an afterthought. Theano is half hearted on 3. Bloomberg doesn't even have a 3 access library (goodbye 3.0 for the entire finance sector). Asyncio, one of the so called killer features of 3, looks great but is 10 years too late and is done much better in other languages (goroutines). Python is seriously losing momentum: hey, even Hacker News quickly demoted this story to page 2 precisely because it attracted the usual flood of 2v3 comments causing the comment number to dwarf the score. It's become a controversial and uncool language thanks to this forced 3 story, and the fact that 6 years later in this dog-years world, this issue is still on the table, makes 3 an utter failure.
I predict that 2.7.9.x.y.z will continue well past 2020 and backport any truly useful stuff from 3 because that's much easier than porting 2 code to 3. Or, and the core developers should seriously think about this, they should build 2->3 right into the interpreter so you can run, unmodified 2.7 code on a 3 interpreter (2.7 libraries included), and then pick and choose the 3 features you want yourself.
I predict that 2.7.9.x.y.z will continue well past 2020 and backport any truly useful stuff from 3 because that's much easier than porting 2 code to 3. Or, and the core developers should seriously think about this, they should build 2->3 right into the interpreter so you can run, unmodified 2.7 code on a 3 interpreter (2.7 libraries included), and then pick and choose the 3 features you want yourself.
> Scientific computing is stuck with 2.
False. NumPy, SciPy, scikit-learn, Pandas, and friends all work great on Python 3.
False. NumPy, SciPy, scikit-learn, Pandas, and friends all work great on Python 3.
It's not a question whether it works. It's a question of whether it's being used. And it's not, really. Those code paths are less tested and less performant, so why would you choose them even if you start a new project without baggage?
Python 3 won't see any traction unless it offers a clear improvement over 2. Some people think that's async. I have my doubts.
Python 3 won't see any traction unless it offers a clear improvement over 2. Some people think that's async. I have my doubts.
I think there is a certain kind of sloppiness in Python culture that holds it back.
For instance, the split between Python 2 and Python 3 has kept from me getting more into Python because I have to choose one or the other in terms of which libraries I want to be compatible with (and the libraries are a big reason I'd want to to use Python.)
If you look at Java or PHP or .NET there is a lot of concern about keeping compatibility between versions, but going from 2 to 3 they changed a lot of APIs that they didn't have to.
The most disturbing thing is that I have been shouted down for years on places like proggit for bringing this issue up but it is only in the last year that I have seen it talked about by pythonistas, and that's a general problem I see with Python, that it ducks the hard questions which Java, in particular, faces before anyone else. (i.e. having a memory model for threads which is mature)
On one hand, the GIL means that Python is not scalable in terms of threads. Today 4 core machines are mainstream, even for phones, and with a language that supports scalable threads, you can get a 4x speedup easily for many real apps.
The other option is go to the Node route, which is balls-to-the-walls async operations in a single process, but (i) the async people also have had a culture of "kicking the can down the road", (ii) it is hard to get programmers who are used to sync APIs to switch to async (this is one reason why the "Modern" interface in Win 8 was DOA... Microsoft was pushing people to use Javascript because rewriting Win32 apps to be async defeats the purpose of using Win32 which is being able to use all the old stuff and not learning anything new.)
For instance, the split between Python 2 and Python 3 has kept from me getting more into Python because I have to choose one or the other in terms of which libraries I want to be compatible with (and the libraries are a big reason I'd want to to use Python.)
If you look at Java or PHP or .NET there is a lot of concern about keeping compatibility between versions, but going from 2 to 3 they changed a lot of APIs that they didn't have to.
The most disturbing thing is that I have been shouted down for years on places like proggit for bringing this issue up but it is only in the last year that I have seen it talked about by pythonistas, and that's a general problem I see with Python, that it ducks the hard questions which Java, in particular, faces before anyone else. (i.e. having a memory model for threads which is mature)
On one hand, the GIL means that Python is not scalable in terms of threads. Today 4 core machines are mainstream, even for phones, and with a language that supports scalable threads, you can get a 4x speedup easily for many real apps.
The other option is go to the Node route, which is balls-to-the-walls async operations in a single process, but (i) the async people also have had a culture of "kicking the can down the road", (ii) it is hard to get programmers who are used to sync APIs to switch to async (this is one reason why the "Modern" interface in Win 8 was DOA... Microsoft was pushing people to use Javascript because rewriting Win32 apps to be async defeats the purpose of using Win32 which is being able to use all the old stuff and not learning anything new.)
You get a 4x speedup with multicore, and get a 100x slowdown for using Python. Furthermore, Python supports multiple cores perfectly well as long as you use multiple processes. Python threads also work fine for I/O. The only thing it doesn't do is shared memory parallelism. Maybe staying within one process gives you an x% speedup compared to the overhead of sending data between processes, but it simply does not make much sense compared to the 100x. If you need that kind of performance you would not use Python.
GIL worries are not founded, most of the time.
Multithreads in Python works just fine for I/O bounded operations. Switch a multithread program to use multiple processes (and avoid GIL issues) is quite simple with the multiprocessing module[1] if you need to achieve 100% CPU usage.
In my experience, complains about the GIL are mostly FUD.
[1]https://docs.python.org/2/library/multiprocessing.html
Multithreads in Python works just fine for I/O bounded operations. Switch a multithread program to use multiple processes (and avoid GIL issues) is quite simple with the multiprocessing module[1] if you need to achieve 100% CPU usage.
In my experience, complains about the GIL are mostly FUD.
[1]https://docs.python.org/2/library/multiprocessing.html
Some languages manage to have a really nice blend between OS threads and lightweight concurrency. Erlang/Elixir can spin up hundreds of thousands of lightweight "processes" (their word for green threads), which get split up between a number of real OS threads by schedulers. It means you can have great performance on IO with green threads, but also have great performance on CPU bound tasks with honest-to-goodness threads.
Clojure's core.async can pull off the same trick, though I don't know as much about how it works.
Clojure's core.async can pull off the same trick, though I don't know as much about how it works.
There are many ways to get around the GIL. Use PyPy. Use processes. And are you really doing that much computation? Threads on standard CPython are perfectly fine for I/O.
PyPy still has a GIL: http://pypy.readthedocs.org/en/latest/faq.html#does-pypy-hav...
Doh. I should have said, "Use Jython."
Yes, I really am doing that much computation. Not everyone works on database-dependent web apps.
The GIL sucks. It holds Python back from wider use. I wanted to use Python for the GUI/scripting layer of a high performance desktop application, but dealing with the GIL in a mixed C++/Python multithreaded event-driven architecture was too annoying. When Python code isn't in charge of the main application loop, things get ugly.
These days, I mainly see Python as a source of syntax inspiration for other languages. I say this with the highest amount of respect. I think Python's best qualities - list comprehensions, generators, itertools - are largely independent of its interpreted, dynamic nature. They will eventually find a better home in a language with more industrial-grade semantics.
"The GIL isn't that bad, get over it" is not an attitude that will sustain a language for decades.
The GIL sucks. It holds Python back from wider use. I wanted to use Python for the GUI/scripting layer of a high performance desktop application, but dealing with the GIL in a mixed C++/Python multithreaded event-driven architecture was too annoying. When Python code isn't in charge of the main application loop, things get ugly.
These days, I mainly see Python as a source of syntax inspiration for other languages. I say this with the highest amount of respect. I think Python's best qualities - list comprehensions, generators, itertools - are largely independent of its interpreted, dynamic nature. They will eventually find a better home in a language with more industrial-grade semantics.
"The GIL isn't that bad, get over it" is not an attitude that will sustain a language for decades.
Numba allows multithreaded gil free numerical code.
To be honest, I think the influence of the Ruby/Rails ecosystem is hurting Python. In the last few years, it's become impossible to run many Python projects without doing a pip install -r requirements.txt. That in itself would not be a problem, except that many pip packages are buggy and poorly maintained. Few people take the time to vet such libraries because the effort involved in a pip install is so low. The result is that we often spend hours configuring a buggy library instead of writing code that would do a better job solving our problems. "Convention over configuration" is no less important than it always was, but it's certainly less fashionable.
There are exceptions. Django rest framework, for example, is an extremely mature, well-made library. But these are becoming less of a percentage of the whole and are harder to find in the chaff.
And there are a few projects still carrying the torch--but typically these are old projects that have been around for a long time, still maintained by older developers who were writing Python when Python was young. We still have at least a decade before these old masters start retiring, so I don't think Python will die any time soon. But it has become more important than ever to be careful what you let into your dependencies.
There are exceptions. Django rest framework, for example, is an extremely mature, well-made library. But these are becoming less of a percentage of the whole and are harder to find in the chaff.
And there are a few projects still carrying the torch--but typically these are old projects that have been around for a long time, still maintained by older developers who were writing Python when Python was young. We still have at least a decade before these old masters start retiring, so I don't think Python will die any time soon. But it has become more important than ever to be careful what you let into your dependencies.
This one resonates with me.
Though I haven't used either myself yet, I've noticed the trend in automated code quality tooling becoming more popular in Python. Specifically, the ones that come to mind are: Codacy, Code Climate (both free for open source); and Landscape.
Personally I'm still using strong classics like coverage and pylint, but plan to explore the new gen stuff soon.
Though I haven't used either myself yet, I've noticed the trend in automated code quality tooling becoming more popular in Python. Specifically, the ones that come to mind are: Codacy, Code Climate (both free for open source); and Landscape.
Personally I'm still using strong classics like coverage and pylint, but plan to explore the new gen stuff soon.
The Python GIL problem could been solved in Python3 by just moving all global interpreter instance variables into a single struct and changing the C API to take a pointer to this interpreter struct as the first argument in all C API functions. No thread local variable hacks need be used and all interpreter instances could have been completely independent of each other, allowing for seamless and efficient share-nothing multi-threading. Python3 was not intended to be 100% compatible with Python2 anyway, so it was a perfect opportunity to fix this long standing design bug. But this opportunity was lost.
Strangely, the Google V8 team repeated Python's API design error and also relies on global interpreter variables and thread-local variables to swap interpreter contexts, or "isolates" in V8-speak. Although my understanding is that the V8 team recognizes the problem and are (slowly) addressing it and changing their APIs accordingly with each successive release.
Strangely, the Google V8 team repeated Python's API design error and also relies on global interpreter variables and thread-local variables to swap interpreter contexts, or "isolates" in V8-speak. Although my understanding is that the V8 team recognizes the problem and are (slowly) addressing it and changing their APIs accordingly with each successive release.
Python is changing the world more than ever before, by becoming the new default.
I know undergrad CS students, masters CS students, non-CS professionals well into their career, and non-CS hobbyists. And they are all writing Python. Five years ago when I was doing my undergrad, that would have been Java, or C++.
It's not exactly an exciting change from a HackerNews viewpoint, but it's pretty amazing for programming as a whole. The people I know are so much more excited to be learning. There is frustration, but much less than when I knew people learning on C++. Obviously that's still a fantastic language, but I'm happy Python is the new default.
I know undergrad CS students, masters CS students, non-CS professionals well into their career, and non-CS hobbyists. And they are all writing Python. Five years ago when I was doing my undergrad, that would have been Java, or C++.
It's not exactly an exciting change from a HackerNews viewpoint, but it's pretty amazing for programming as a whole. The people I know are so much more excited to be learning. There is frustration, but much less than when I knew people learning on C++. Obviously that's still a fantastic language, but I'm happy Python is the new default.
I was excited when MIT switched to Python for setting the trend for other schools to follow that it's okay to teaching computing from the top down.
Python is a very nice language, but a big stumbling block is the poor package management. Contrasting this with node, which makes it as easy as possible to build modular software, python's packages are a big step backwards.
I'm not following this. Can you give examples of features in node?
Given, the quality of things in any community repo, and the namespacing, are often terrible, but this hasn't ever kept me from using good things out of it.
I'm also not sure how it can be a step backwards, as Python is quite a bit older :)
One thing I don't like about npm is folks are starting to store non-npm things in it, which I think may because of the weird situation of npm getting funding. I don't understand that just as much as I wouldn't understand PyPi or CPAN getting funding :)
Given, the quality of things in any community repo, and the namespacing, are often terrible, but this hasn't ever kept me from using good things out of it.
I'm also not sure how it can be a step backwards, as Python is quite a bit older :)
One thing I don't like about npm is folks are starting to store non-npm things in it, which I think may because of the weird situation of npm getting funding. I don't understand that just as much as I wouldn't understand PyPi or CPAN getting funding :)
The key difference between pypi and npm is that if you pull a package down and it depends on something==1.0.2 and you depend directly on the incompatible something==0.8.9 both version live together in peace because only request will use 1.0.2. With pypi (and most other dependency management systems) you'll have to make a choice on which one you want to use and in this example you'll have to choose an older version of the library depending on something==1.0.2.
This has been largely solved with virtualenvs where those collisions happen on the system level, but I concede it's harder when the mutually incompatible requirements are in the same project.
If I'm understanding the question correctly, I agree virtualenv + virtualenvwrapper solves it mostly. For what I consider a decent treatment of some of the shortcomings in pip vs. npm, I found this post helpful:
"Things I wish Pip learned from Npm" https://medium.com/@alonisser/things-i-wish-pip-learned-from...
"Things I wish Pip learned from Npm" https://medium.com/@alonisser/things-i-wish-pip-learned-from...
I've done just a small amount of work with node, but as a professional Python dev, I very much agree that package management is one of (if not the most) significant downsides of our ecosystem.
One feature I'd love to have personally is a clear way to manage only direct dependencies, but have reproducible builds in pip. @nvie has done some great work in this space [0][1], but not all of us have caught up yet. :)
0: http://nvie.com/posts/pin-your-packages/
1: http://nvie.com/posts/better-package-management/
One feature I'd love to have personally is a clear way to manage only direct dependencies, but have reproducible builds in pip. @nvie has done some great work in this space [0][1], but not all of us have caught up yet. :)
0: http://nvie.com/posts/pin-your-packages/
1: http://nvie.com/posts/better-package-management/
anaconda solves this
Im currently migrating a large FTSE100 web application stack from Java/SpringMVC over to Python/Django.. Its been a long journey (4 years!) to unhinge Java as the given option, but Pythons proven time and time again to save millions per project, and often with better results because developers can concentration on product rather than framework problems.
The focus is on getting things done, today and tomorrow; not be trendy or be on the first page of HN or disrupt the language space using a new paradigm.
Python is "silently?" used A LOT. Not only on web development (which has more the spotlight on the Internet) but in areas like systems scripting, data analysis or science research. I think that is a great language with a lot of opportunities (including career opportunities) that is still growing.
Yes, it's not the new kid in town. I think of it more as an advantage, really.