On Learning Rust and Go: Migrating Away from Python(blog.liw.fi)
blog.liw.fi
On Learning Rust and Go: Migrating Away from Python
https://blog.liw.fi/posts/2019/03/24/on_learning_rust_and_go_migrating_away_from_python/
337 comments
I'm going to have a go with Rust and Go too, but for different reasons.
IMO Python is still the best language to go with when you are writing anything "scripty":
- Stuff that's not expected to be long. Do one thing well, let some other program do another, compose the higher order functions from functions that work. Of course this isn't always possible.
- Stuff where the types are not going to confuse you. Often you just have a few types, and it's fairly obvious what you can do with a given object. If you get too many types, there's a good chance it will start to get confusing and then strong typing would help.
- Stuff where performance is not a problem. Don't write your HFT system in Python. You simulation though, might benefit from orchestration of some VMs.
The ultimate use case for Python is glue code. And guess what, a lot of code is just glue. A good glue language has a large ecosystem of libs, and some language features that make it easy to read. That's Python in a nutshell: whitespace formatting makes things easy to read, and you get built in list/dict syntax to keep everything short and sweet.
For me Rust and Go are interesting having come from C++. There's a lot of pitfalls in C++ that might be helped with Rust. At the moment I have a whole load of Valgrind suite tests that look for potential problems. Perhaps Rust can bake those into the language? I haven't had time to look at it yet.
Go seems to be interesting as a compromise between easy and fast, and by the sound of it things have turned out well. I have people on my team using it, and it seems to be the thing to do for web services that need to be faster than nodejs but you don't want to spend a load of time making sure a c++ version is correct.
IMO Python is still the best language to go with when you are writing anything "scripty":
- Stuff that's not expected to be long. Do one thing well, let some other program do another, compose the higher order functions from functions that work. Of course this isn't always possible.
- Stuff where the types are not going to confuse you. Often you just have a few types, and it's fairly obvious what you can do with a given object. If you get too many types, there's a good chance it will start to get confusing and then strong typing would help.
- Stuff where performance is not a problem. Don't write your HFT system in Python. You simulation though, might benefit from orchestration of some VMs.
The ultimate use case for Python is glue code. And guess what, a lot of code is just glue. A good glue language has a large ecosystem of libs, and some language features that make it easy to read. That's Python in a nutshell: whitespace formatting makes things easy to read, and you get built in list/dict syntax to keep everything short and sweet.
For me Rust and Go are interesting having come from C++. There's a lot of pitfalls in C++ that might be helped with Rust. At the moment I have a whole load of Valgrind suite tests that look for potential problems. Perhaps Rust can bake those into the language? I haven't had time to look at it yet.
Go seems to be interesting as a compromise between easy and fast, and by the sound of it things have turned out well. I have people on my team using it, and it seems to be the thing to do for web services that need to be faster than nodejs but you don't want to spend a load of time making sure a c++ version is correct.
> Note that I've not written any significant code in either language, so I'm just writing based on what I've learnt by reading.
So he’s basically comparing 20+ years of Python usage with marketing material from Go and Rust. Right.
I wish people could be honest with their motivations, instead of making up excuses to justify this sort of change. Here it’s a classic case of “I got bored and I don’t like the new features, want new shiny”, wrapped in a bit of whining about problems that have well-known solutions (use a good IDE, use type hints or one of the many libs that enforce types...). I have no problem with that, just don’t try to sell it to me on technical grounds, because it looks very much like you have none.
Python is still one of the best ways to get stuff done, quickly and with a minimum of boilerplate. Its limitations are well-known and have not really increased for about 15 years - if anything they have shrunk in many, many areas. Adoption rates keep exploding, despite a lot of doom forecasted by some alpha geeks about 10 years ago: “There is no mobile story! We will all be using Ruby/Go/NodeJS tomorrow! Nobody will ever go to Py3!” Etc etc etc.
You got bored by it? That’s fine, it happens, no harm done. But you don’t need to convince anyone, including yourself, that you’re moving on because of this or that flimsy excuse.
So he’s basically comparing 20+ years of Python usage with marketing material from Go and Rust. Right.
I wish people could be honest with their motivations, instead of making up excuses to justify this sort of change. Here it’s a classic case of “I got bored and I don’t like the new features, want new shiny”, wrapped in a bit of whining about problems that have well-known solutions (use a good IDE, use type hints or one of the many libs that enforce types...). I have no problem with that, just don’t try to sell it to me on technical grounds, because it looks very much like you have none.
Python is still one of the best ways to get stuff done, quickly and with a minimum of boilerplate. Its limitations are well-known and have not really increased for about 15 years - if anything they have shrunk in many, many areas. Adoption rates keep exploding, despite a lot of doom forecasted by some alpha geeks about 10 years ago: “There is no mobile story! We will all be using Ruby/Go/NodeJS tomorrow! Nobody will ever go to Py3!” Etc etc etc.
You got bored by it? That’s fine, it happens, no harm done. But you don’t need to convince anyone, including yourself, that you’re moving on because of this or that flimsy excuse.
I'm currently migrating a similarly sized Python and Django application to Go but for different reasons.
For me it's bit rot. My Python is approaching EOL on this version and needs upgrading to Python 3, and I have a lot of 3rd party code in here that made things simpler to create but over time they've changed enough or disappeared that I now have to take on work there too. Then Django and it's internals have changed a lot in the last 6 years.
My Go code written at the same time (6+ years ago) hasn't had any of it's libraries change so much that I've needed to do anything. Every now and then I pull the next major version of Go and recompile and it's done.
This is all side project stuff but a lot of people use it (monthly average visitors being +250k)... it's non-profit and I don't wish to have to maintain it much.
It worked once, if nothing has changed in the functionality it should keep working... but this isn't as true in non-compiled languages where you need to depend on so much of the environment, let alone build culture that doesn't fully vendor dependencies.
I like Python, but prefer to write code and move forward with other projects.
As to Rust vs Go. I code in both and only picked Go here because a lot of the code in another project I worked on can be used here. I'd probably try Rust if I was starting from scratch now. But I'd be a little concerned bit rot would affect Rust too, it's still quite early for the wider ecosystem.
For me it's bit rot. My Python is approaching EOL on this version and needs upgrading to Python 3, and I have a lot of 3rd party code in here that made things simpler to create but over time they've changed enough or disappeared that I now have to take on work there too. Then Django and it's internals have changed a lot in the last 6 years.
My Go code written at the same time (6+ years ago) hasn't had any of it's libraries change so much that I've needed to do anything. Every now and then I pull the next major version of Go and recompile and it's done.
This is all side project stuff but a lot of people use it (monthly average visitors being +250k)... it's non-profit and I don't wish to have to maintain it much.
It worked once, if nothing has changed in the functionality it should keep working... but this isn't as true in non-compiled languages where you need to depend on so much of the environment, let alone build culture that doesn't fully vendor dependencies.
I like Python, but prefer to write code and move forward with other projects.
As to Rust vs Go. I code in both and only picked Go here because a lot of the code in another project I worked on can be used here. I'd probably try Rust if I was starting from scratch now. But I'd be a little concerned bit rot would affect Rust too, it's still quite early for the wider ecosystem.
Having used all three of those languages, I've commented on this before.
Go is mediocre as a a language, but it has one big edge - if you're doing web backend stuff, the libraries for that are the ones used internally by Google. So they're being executed billions of times per second on Google servers and tend to be well behaved. Python tends to have multiple libraries for basic functions like the database interface, all with different bugs. If you write web backend stuff in Go, it will probably work.
Having suffered through the Python 2 to 3 transition, I'm less of a Python fan than I used to be. At some point, they need to stop adding features. Currently, Python is acquiring unchecked "type hints", which look like a terrible idea. Optional typing, sure, but unchecked type hints? If the compiler enforced them, they might be worth something. (And no, running a separate type checking tool is not the answer. Especially since no such tool exists that's production grade and in sync with the language.)
Rust has no idea when to stop adding features. Rust started as an imperative language and then morphed into a semi-functional language. Mostly to get a sane error handling model. The borrow checker was a brilliant innovation, though. I used Rust for a while, but quit about two years ago in disgust. I never want to hear "that only compiles in nightly" again.
Go is mediocre as a a language, but it has one big edge - if you're doing web backend stuff, the libraries for that are the ones used internally by Google. So they're being executed billions of times per second on Google servers and tend to be well behaved. Python tends to have multiple libraries for basic functions like the database interface, all with different bugs. If you write web backend stuff in Go, it will probably work.
Having suffered through the Python 2 to 3 transition, I'm less of a Python fan than I used to be. At some point, they need to stop adding features. Currently, Python is acquiring unchecked "type hints", which look like a terrible idea. Optional typing, sure, but unchecked type hints? If the compiler enforced them, they might be worth something. (And no, running a separate type checking tool is not the answer. Especially since no such tool exists that's production grade and in sync with the language.)
Rust has no idea when to stop adding features. Rust started as an imperative language and then morphed into a semi-functional language. Mostly to get a sane error handling model. The borrow checker was a brilliant innovation, though. I used Rust for a while, but quit about two years ago in disgust. I never want to hear "that only compiles in nightly" again.
Strongly agree with author. I really like Python and at one point thought that it makes sense to write almost any project in it, since it's so versatile and I know it pretty well.
After working on a quite large application and having to use lots of assert(isinstance(arg, type) at the beginning of almost every function, I began to think that a strong type system is very much needed for large projects. I believe this was one of the reasons why Twitter moved from Ruby to Scala.
I still love to use Python for hacking something quick for myself. But I also look at some popular strongly-typed languages now and hope to get better at one of them soon.
After working on a quite large application and having to use lots of assert(isinstance(arg, type) at the beginning of almost every function, I began to think that a strong type system is very much needed for large projects. I believe this was one of the reasons why Twitter moved from Ruby to Scala.
I still love to use Python for hacking something quick for myself. But I also look at some popular strongly-typed languages now and hope to get better at one of them soon.
Thanks for The writeup, I am going through a similar process. I just retired last Friday (I am an old guy) from managing a machine learning team and everything was done in Python.
For personal projects (and past consulting work) I have been loving Lisp languages since the 1980s but many languages have so many of the benefits of Lisp (thinking of Haskell and Julia) and there is definitely a ‘lisp tax’ on deploying and shipping products.
Anyway, for writing new books, writing open source projects, and perhaps some consulting, I would like to cut back to a single language, instead of the 4 or 5 I use now.
EDIT: I did some Go Lang codelabs when I worked at Google and have played with Rust (principally spending some evenings with an open source blockchain project) and neither language will be my home base language.
For personal projects (and past consulting work) I have been loving Lisp languages since the 1980s but many languages have so many of the benefits of Lisp (thinking of Haskell and Julia) and there is definitely a ‘lisp tax’ on deploying and shipping products.
Anyway, for writing new books, writing open source projects, and perhaps some consulting, I would like to cut back to a single language, instead of the 4 or 5 I use now.
EDIT: I did some Go Lang codelabs when I worked at Google and have played with Rust (principally spending some evenings with an open source blockchain project) and neither language will be my home base language.
> Rust is developed by a community, and was started by Mozilla. Go development seems to be de facto controlled by Google, who originated the language. I'd rather bet my non-work future on a language that isn't controlled by a huge corporation, especially one of the main players in today's surveillance economy.
Anyone else agree with this view ? Programming languages should be choosen based on technical merits, rather than who is behind it. Is there a lesson from history that I am missing ?
Anyone else agree with this view ? Programming languages should be choosen based on technical merits, rather than who is behind it. Is there a lesson from history that I am missing ?
The strengths of Go become apparent when working in teams and when deploying, not when reading blogs about the language before trying to write code in it.
I'm a long time C++ developer, and recently took a job where I write a lot of Python in a large code base. Initially it seemed great, but I quickly realized how uncomfortable it is to have no safety net up front. The interesting thing is Python eliminates a lot of programming errors you may encounter in C++. But it introduces a whole different set of programming errors you would never see in C++. I don't feel comfortable using Python for any project where the code must be correct. I would use Rust every time for that.
Not trying to dismiss this person's work in any way, but is there any reason why this blog post should be considered notable? They clearly say at the end that they've yet to write any real-world code in either Rust or Go, and so far have just been learning about either language - so how come that this shoots up to the top of HN? Is it just the magic keywords "Rust" and "Go" that do that?
At 15000 loc, if the author found it difficult to organize python code, then it's not python's fault.
I find it a pleasure to organize code in python.
What am I missing here?
I find it a pleasure to organize code in python.
What am I missing here?
I have a genuine question. One of Rust's biggest plus points is having no GC and the related overhead at runtime. Go, on the other hand does use GC. Then how come so many successful Systems programs (e.g. Kubernetes) are written in Go. Anything I am missing here?
For me I think it comes down to what is the right tool for the job. I have been using Python for years for web scraping & data analysis (via Scrapy and SciPy) and I feel that they are the best tools for what I am doing.
Now that a lot of my scraping needs require me to render the DOM, I have had to run a headless web browser (previously PhantomJS/Splash and now Chrome). Chrome released the Puppeteer library which is the most straight forward way to interact with Headless Chrome. This is in Node so that is what i’m using.
On my last project we needed to parse through a large amount of HTML files to extract relevant information (about 70,000,000) and put it in a DB for further analyses. I have done similar things in Python before, but there are blocking/concurrency issues that can make it slow and frustrating to build.
My solution was to just do this in Go. I can then run my extraction in a goroutine. Go’s selling point for me was speed (relative to Python) and dead simple concurrency - which made it the right tool for the job.
Edit: Spelling, iOS’s autocorrect can be so annoying sometimes :)
Now that a lot of my scraping needs require me to render the DOM, I have had to run a headless web browser (previously PhantomJS/Splash and now Chrome). Chrome released the Puppeteer library which is the most straight forward way to interact with Headless Chrome. This is in Node so that is what i’m using.
On my last project we needed to parse through a large amount of HTML files to extract relevant information (about 70,000,000) and put it in a DB for further analyses. I have done similar things in Python before, but there are blocking/concurrency issues that can make it slow and frustrating to build.
My solution was to just do this in Go. I can then run my extraction in a goroutine. Go’s selling point for me was speed (relative to Python) and dead simple concurrency - which made it the right tool for the job.
Edit: Spelling, iOS’s autocorrect can be so annoying sometimes :)
This is more of a general observation but why are Rust and Go so often mentioned in the same sentence? They are designed around very different use cases such that a direct comparison doesn't seem to warrant as much attention as it seems to get.
15K LOC in Python. Be prepared to increase your LOCs to at least 33-100% when writing in Rust or Go, hopefully your error rate by line of code will decrease more than your overall LOC.
There are not only benefits.
Folks may also be interested in a thorough analysis of porting a big Python app (0install) to OCaml, after considering various alternatives like Rust and Haskell: http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-ret...
The author's series on learning Rust, not linked in the OP https://blog.liw.fi/learning-rust/
I really enjoy the author's blog, which has been discussed here many times over the years https://hn.algolia.com/?query=liw.fi
I really enjoy the author's blog, which has been discussed here many times over the years https://hn.algolia.com/?query=liw.fi
Every time reading someone changing from xxx to Rust feel the need to post this thread, it's really fun indeed.
https://www.reddit.com/r/rust/comments/acjcbp/rust_2019_beat...
https://www.reddit.com/r/rust/comments/acjcbp/rust_2019_beat...
I share the author's sentiment, and it's what lead me to Haskell. It's faster and cheaper for me to write prototypes in Haskell than it ever was in Ruby.
And at least in startup-world which is the bulk of my career, the speed at which you can build a reasonably robust prototype can be the life or death of a business, and this prototype usually ends up being the production product anyway. Prototypes are seldom rewritten "properly".
And at least in startup-world which is the bulk of my career, the speed at which you can build a reasonably robust prototype can be the life or death of a business, and this prototype usually ends up being the production product anyway. Prototypes are seldom rewritten "properly".
After all python is still an interpreted scripting language. To me comparing it to rust and go is like apples and oranges.
I have reverse experience I've used pretty much everything under the sun, but never really taken Python seriously just had to do a project in Python @ work and it was such a pleasant experience so my current mindset Python + Cython for most projects unless things are really perfomance sensitive than Rust.
Line of code is a relative measure. If you leverage package like numpy/pandas, asyncio and co, you could easily write python code that are ten fold shorter than their C/rust/go equivalent. And improving speed when you make a good use of numpy requires quite an amount of time and skills.
I am personally investing heavily in Rust. I hope to bear some fruit
Go and Rust as substitutes for python, like really? If it works for you perfect, but this is not serious even for a debate, generally speaking, for the majority of programmers
I like that you're approaching both Rust and Go, instead of just picking one. Because a _war_ between the two on which is best seems to be quite common.
"Partly it's because I'd like more help from my programming tools, such as static type checking, better handling of abstractions"
As soon as I read that, I know immediately that I'm dealing with an object-oriented programmer. Always the same gripes, always the same problems! Your programming paradigm is busted and no language will ever be able to change that, no matter what lexicography and grammar it implements! There is a reason why procedural or functional programmers don't complain about those types of issues!
As soon as I read that, I know immediately that I'm dealing with an object-oriented programmer. Always the same gripes, always the same problems! Your programming paradigm is busted and no language will ever be able to change that, no matter what lexicography and grammar it implements! There is a reason why procedural or functional programmers don't complain about those types of issues!
try java.
no, seriously.
java and a good IDE (netbeans or eclipse, not some editors like vscode) are a godsend to handle "bigger" project.
It's absolutely fast to compile and insanely on-the-fly static checks all over the place.
So a typically unimaginative python programmer sees new Kool aid and is thinking about taking a drink. Decides to write blog post about it. And somehow it's #1 on hn.
Python has all the bells and whistles you need to build large code bases, but you do need to get organized, no language can really do that for you and attempts at forcing you to do it give rise to 'factory factory' nonsense. I'm no huge fan of Python, I use it when I have to but Rust is from a bells-and-whistles point of view a step down from Python, it is more a better 'C' than a better Python and Go has it's own share of issues that has kept me away from it so far.
If you want to crank out some research project or application quickly building on the libraries that others provide (especially in machine learning or other data intensive fields) then Python would be my first choice, and something like 'R' probably my second (though I see 'R' mostly as a data scientist's workbench that has a programming language bolted on).