On Distributing Command Line Applications: Why I Switched From Ruby to Go(codegangsta.io)
codegangsta.io
On Distributing Command Line Applications: Why I Switched From Ruby to Go
http://codegangsta.io/blog/2013/07/21/creating-cli-applications-in-go/
73 comments
>For me, this is worse. There's no audit trail. No way for me to verify that the binary of your app is the same as the source you provide. So there's no way I'm running your binary on my system. I simply cannot trust it with my user account.
It's not for you then.
For one, billions of people run tons of binary third party apps all day long.
Second, when you deploy to people, they have to trust you. Whether they trust your code is irrelevant. Most of them are non-programmers anyway and wouldn't even check if you gave them a source code printout.
It's not for you then.
For one, billions of people run tons of binary third party apps all day long.
Second, when you deploy to people, they have to trust you. Whether they trust your code is irrelevant. Most of them are non-programmers anyway and wouldn't even check if you gave them a source code printout.
Nothing is stopping you from building the app from source instead, just like every other open source project. But at least with Go you have the _option_ of just downloading and running a binary if you trust the author.
This is true. I do have that option.
But I very rarely trust the author. Most of the time I've never heard of the author. This is a good thing, and exactly why Github has been so successful. By nature, if there's something obscure that I need doing, and I find some software or software module that lets me do it, then chances are I won't know the author.
This means that downloading and running the binary is suddenly a very obscure use case. It should be for you, too.
But I very rarely trust the author. Most of the time I've never heard of the author. This is a good thing, and exactly why Github has been so successful. By nature, if there's something obscure that I need doing, and I find some software or software module that lets me do it, then chances are I won't know the author.
This means that downloading and running the binary is suddenly a very obscure use case. It should be for you, too.
> This means that downloading and running the binary is suddenly a very obscure use case. It should be for you, too.
A rare use case for you, but definitely not an obscure one in general. Besides, trusting the author, in this case, only matters if you're willing to audit the source to verify that nothing malicious is occurring -- something I'm not going to do in general. I mean, realistically, when was the last time you audited your network drivers? your OS? your browser?
A rare use case for you, but definitely not an obscure one in general. Besides, trusting the author, in this case, only matters if you're willing to audit the source to verify that nothing malicious is occurring -- something I'm not going to do in general. I mean, realistically, when was the last time you audited your network drivers? your OS? your browser?
You don't have to verify the source yourself. Others can do it. I trust the Debian maintainer for my package. He's a real world person who has had his key signed by other Debian developers.
The fact that the source can be verified also reduces the risk of malicious modification.
It's not necessarily just the author who might be malicious, either. The author's computer where he runs his compiler might be compromised.
The fact that the source can be verified also reduces the risk of malicious modification.
It's not necessarily just the author who might be malicious, either. The author's computer where he runs his compiler might be compromised.
Do you trust your compiler?
"You can't trust code that you did not totally create yourself." - Ken Thompson[1].
[1] "Reflections on Trusting Trust", Communication of the ACM, Vol. 27, No. 8, August 1984, pp. 761-763, http://cm.bell-labs.com/who/ken/trust.html
"You can't trust code that you did not totally create yourself." - Ken Thompson[1].
[1] "Reflections on Trusting Trust", Communication of the ACM, Vol. 27, No. 8, August 1984, pp. 761-763, http://cm.bell-labs.com/who/ken/trust.html
Could you expand on what you mean by "I very rarely trust the author"? Isn't using their work a statement of trust?
Source or binary, you are still embracing their description of tasks for an automated machine to perform very rapidly.
If you are wary of what they have done, it's true that source is somewhat more accessible to examine than machine code, but for any non-trivial program source is still quite expensive to evaluate.
Source or binary, you are still embracing their description of tasks for an automated machine to perform very rapidly.
If you are wary of what they have done, it's true that source is somewhat more accessible to examine than machine code, but for any non-trivial program source is still quite expensive to evaluate.
I don't necessarily need to trust the author directly. If he provides source, and the source is known to be the true source of the binary I'm using, then the source can be checked by others, even after a problem has been detected.
> Isn't using their work a statement of trust?
Trust isn't absolute. I might trust an author's intentions, but not that his computer has not been compromised, for example.
Being forced to provide source could be argued to keep authors honest, too.
> Isn't using their work a statement of trust?
Trust isn't absolute. I might trust an author's intentions, but not that his computer has not been compromised, for example.
Being forced to provide source could be argued to keep authors honest, too.
> This means that downloading and running the binary is suddenly a very obscure use case. It should be for you, too.
That's very much not true, especially when you start dealing with corporate/enterprise systems (and remember, Go is intended as a systems programming language).
Just last week I heard Eliot Horowitz (CTO of 10Gen) talking about how it's advantageous for them to use Go for parts of their product, because they can provide guarantees about the software that they're running ("Hey, you're running the exact same file as what we're providing you, not fetching dependencies separately and interpreting things on-the-fly, etc.")
From a security perspective, these corporate clients have already entered into contractual agreements where they don't really care to verify that the source and binary refer to the same code.
If I'm paying a third-party company to provide me with software and services, I'm trusting them to provide security at multiple levels. I'm probably not even auditing their source code myself, so the idea that they would ship us a binary that didn't match the source code is something that wouldn't even enter my mind.
What I've described above is not an obscure use case.
Yes, there are other things that could go wrong that statically compiled binaries can't solve, but from the point of view of a bank or a large, non-tech corporation, it's one less link that could fail in the process.
That's very much not true, especially when you start dealing with corporate/enterprise systems (and remember, Go is intended as a systems programming language).
Just last week I heard Eliot Horowitz (CTO of 10Gen) talking about how it's advantageous for them to use Go for parts of their product, because they can provide guarantees about the software that they're running ("Hey, you're running the exact same file as what we're providing you, not fetching dependencies separately and interpreting things on-the-fly, etc.")
From a security perspective, these corporate clients have already entered into contractual agreements where they don't really care to verify that the source and binary refer to the same code.
If I'm paying a third-party company to provide me with software and services, I'm trusting them to provide security at multiple levels. I'm probably not even auditing their source code myself, so the idea that they would ship us a binary that didn't match the source code is something that wouldn't even enter my mind.
What I've described above is not an obscure use case.
Yes, there are other things that could go wrong that statically compiled binaries can't solve, but from the point of view of a bank or a large, non-tech corporation, it's one less link that could fail in the process.
OK, I'll give you that. In an enterprise environment, a company can choose to trust specific vendors' binaries, and this often happens.
However, I don't think this has much bearing on the original pain point. I'm familiar with these sorts of production environments. Running a local store of gems is standard practice already. There is little marginal cost to doing this.
However, I don't think this has much bearing on the original pain point. I'm familiar with these sorts of production environments. Running a local store of gems is standard practice already. There is little marginal cost to doing this.
Hacker News hosted the following discussion regarding problems that have already occurred maintaining Go software due to reliance on binary distribution: "Golang dependency hell in Haunts The Manse game" https://news.ycombinator.com/item?id=5796597
In this specific instance it was more of an incorrect understanding of Go dependencies and misuse/lack of source control, but it seemed to at least come close to an echo of the issues you are raising. The specific quote from the comments: "There is no version control for dependencies".
In this specific instance it was more of an incorrect understanding of Go dependencies and misuse/lack of source control, but it seemed to at least come close to an echo of the issues you are raising. The specific quote from the comments: "There is no version control for dependencies".
That wasn't a binary distribution problem or anything like one, was it? That was as I remember it a development team that simply hadn't created a known-good repository of their third-party dependencies. While I grant you that's a common pitfall in the rubygem era, it's hard to feel much sympathy as someone who came up building commercial C code, where not having a one-command third-party build-world system was a cardinal sin.
Create a known-good repository of third-party dependencies is so easy in Go. Change the $GOPATH, `go get` all the dependencies and add that folder to version control.
I think I agree. Also, for all the complaints about "magic" in Rails, there's nothing more magical in both Rubyland and Pythonistan than package management, and the explicitness of Golang's "package" system is a welcome contrast.
> there's nothing more magical in both Rubyland and Pythonistan than package management,
Could you expand on this a bit? I'm interested in your opinions, as it's hard to see from the inside. In Ruby, it's just things modifying the $LOAD_PATH, which should be basically the same as changing your $GOPATH, no?
Could you expand on this a bit? I'm interested in your opinions, as it's hard to see from the inside. In Ruby, it's just things modifying the $LOAD_PATH, which should be basically the same as changing your $GOPATH, no?
I don't know much / anything about ruby package management, but if it's commonplace for ruby packages to change the LOAD_PATH (which I guess is similar to PYTHON_PATH / sys.path in python ?), that is really quite magical (by which I mean, causes unexpected behavior).
What conroy is talking about is creating a static set of dependencies that you check into your own source control, because go doesn't let you specify e.g. github commit hashes to "pin" a package dependency to a certain revision.
So a workaround for that is to (as a one-off) change your GOPATH ($GOPATH/src and $GOPATH/bin is where the standard "go get" / "go install" etc put things) to something temporary, "go get" your dependences (which will fetch them from github or wherever at the HEAD revision). Now run your test suite to make sure everything passes. If it does, great, take the current $GOPATH/src and check the contents of that into your own revision control. Et voila, ghetto-"pin"ning.
What conroy is talking about is creating a static set of dependencies that you check into your own source control, because go doesn't let you specify e.g. github commit hashes to "pin" a package dependency to a certain revision.
So a workaround for that is to (as a one-off) change your GOPATH ($GOPATH/src and $GOPATH/bin is where the standard "go get" / "go install" etc put things) to something temporary, "go get" your dependences (which will fetch them from github or wherever at the HEAD revision). Now run your test suite to make sure everything passes. If it does, great, take the current $GOPATH/src and check the contents of that into your own revision control. Et voila, ghetto-"pin"ning.
The packages themselves don't change the LOAD_PATH. rubygems/bundler do. Both explicitly add the paths given in the gemspec (lib/ by default). I wouldn't call that "magical".
Yes, you would do the exact same thing: set your $GEM_HOME and gem install, or use 'bundle install --path.' It'd be basically the same thing.
Not just command line apps. Any time you need to deploy on multiple machines, Go saves you a lot of trouble.
At work we use primarily `buildout` [1] to deploy Python apps with complex dependencies. It is rather slow and fragile. Additionally, PyPi and various external repositories are not very reliable, and we ended up setting up local mirrors. Nevertheless, the process is not very pleasant.
We are experimenting with Go in some smaller projects, and so far the experience is great! We can build on a development machine (OS X) with all the dependencies version controlled (using git-subtree) along with our code, and cross-compile a static binary for Linux for deployment. Deploying is simply copying the binary to the production machines (usual configuration management is done via Puppet).
Besides, Go can efficiently make use of multiple cores (remember to set GOMAXPROCS correctly), and we can just deploy one instance of an app per server, unlike one instance per core as in Python's case (I know there's multiprocessing, but it's bad for operation and management). This greatly reduces memory overhead if the app is big (e.g. consumes a couple hundred of MB upfront and you have 32 cores per machine = a few GB's memory is wasted).
Only problem is that Go's 3rd-party libs are not as comprehensive as Python's yet, and many packages we rely on do not have mature/reliable counterparts in Go. So we are still using Python for our core stuff, but smaller projects that do not require many external packages can be done in Go elegantly.
[1]: http://www.buildout.org/en/latest/
At work we use primarily `buildout` [1] to deploy Python apps with complex dependencies. It is rather slow and fragile. Additionally, PyPi and various external repositories are not very reliable, and we ended up setting up local mirrors. Nevertheless, the process is not very pleasant.
We are experimenting with Go in some smaller projects, and so far the experience is great! We can build on a development machine (OS X) with all the dependencies version controlled (using git-subtree) along with our code, and cross-compile a static binary for Linux for deployment. Deploying is simply copying the binary to the production machines (usual configuration management is done via Puppet).
Besides, Go can efficiently make use of multiple cores (remember to set GOMAXPROCS correctly), and we can just deploy one instance of an app per server, unlike one instance per core as in Python's case (I know there's multiprocessing, but it's bad for operation and management). This greatly reduces memory overhead if the app is big (e.g. consumes a couple hundred of MB upfront and you have 32 cores per machine = a few GB's memory is wasted).
Only problem is that Go's 3rd-party libs are not as comprehensive as Python's yet, and many packages we rely on do not have mature/reliable counterparts in Go. So we are still using Python for our core stuff, but smaller projects that do not require many external packages can be done in Go elegantly.
[1]: http://www.buildout.org/en/latest/
He leaves out discussion of a crucial part of app distribution: dynamic libraries.
While you can't create and link with dynamic libraries in Go, you can call into C or C++ libraries. Go binaries are self-contained as long as all your code is Go. But if you're depending on a C or C++ library, you still have to worry about this aspect of distribution.
While you can't create and link with dynamic libraries in Go, you can call into C or C++ libraries. Go binaries are self-contained as long as all your code is Go. But if you're depending on a C or C++ library, you still have to worry about this aspect of distribution.
It is now possible to statically compile+link C libraries into your Go binary. The sqlite3 package (http://godoc.org/code.google.com/p/go-sqlite/go1/sqlite3) is a good example of this.
Another aspect is that some opensource licensing schemes may require you to link dynamically. The LGPL, for example, requires you to do this unless you release the rest of your source code.
I'd have thought you could just statically link with the C and C++ libraries, creating a static binary?
edit: beaten by rwcarlsen
edit: beaten by rwcarlsen
Blog author here. Long story short I got frustrated with distributing Ruby cli apps and decided to switch to Go for speed and distribution.
I also created a nice cli library built on top of the Go flag package. I've been having a lot of fun in Go so far and encourage anyone who hasn't given it a shot to try it out.
https://github.com/codegangsta/cli
I also created a nice cli library built on top of the Go flag package. I've been having a lot of fun in Go so far and encourage anyone who hasn't given it a shot to try it out.
https://github.com/codegangsta/cli
Made the same decision as you recently.
The only pep-peeve of mine is that I'm not a huge fan of using JSON as a configuration mechanism. I much preferred having a Ruby file for configuration. My middle ground here is using a system ruby via /usr/bin/env ruby and generating the necessary JSON output for people that want the nice syntax.
The only pep-peeve of mine is that I'm not a huge fan of using JSON as a configuration mechanism. I much preferred having a Ruby file for configuration. My middle ground here is using a system ruby via /usr/bin/env ruby and generating the necessary JSON output for people that want the nice syntax.
This also makes go applications trivial to deploy. What used to be a massive amount of plumbing and bootstrapping for a ruby environment with multiple app server instances behind a reverse proxy with all the associated configuration and dependencies can now be little more than pushing a binary to a (nearly) stock machine instance. It really simplifies ops.
I really don't see what having multiple app server instances behind reverse proxy has anything to do with Ruby vs Go. App servers behind reverse proxy is an architectural decision. You can write a Go web app that must be put behind a reverse proxy, or you can write a Go web app that is supposed to act as its own web server. Likewise, your Ruby app does not have to run multiple app server instances behind a reverse proxy. You can run it together with your main web server (Apache or Nginx) and you can run a single instance instead. See https://www.phusionpassenger.com/
Of course, installing dependencies still matters. For Ruby, you can just use the Brightbox Debian/Ubuntu packages.
Of course, installing dependencies still matters. For Ruby, you can just use the Brightbox Debian/Ubuntu packages.
All the mainstream ruby web frameworks require something like nginx in front of multiple instances of something like thin. You can use something like phusion, unicorn or rainbows that do forking for you instead of launching multiple processes by hand, but you still have to configure them and they're still running multiple app instances.
Go's built-in http library, and all the associated frameworks, run multi-threaded out of the box, and are fast enough that you don't need to put nginx or haproxy in front of them.
I love ruby, I've been building and deploying rails apps since 1.0, the ease of deploying go apps is precisely the type of thing the Ruby/Rails community cares a lot about: making developers' lives easier.
Go's built-in http library, and all the associated frameworks, run multi-threaded out of the box, and are fast enough that you don't need to put nginx or haproxy in front of them.
I love ruby, I've been building and deploying rails apps since 1.0, the ease of deploying go apps is precisely the type of thing the Ruby/Rails community cares a lot about: making developers' lives easier.
It is not true that they must run multiple instances. Rails has been multithreading-capable for years. Other Ruby frameworks have been multithreading-capable for much longer. With Phusion Passenger Enterprise, not only do you not have to setup reverse proxying, the sole instance can run multithreaded.
As for "fast enough that you don't need to put nginx or haproxy in front of them" - the point of reverse proxying is not performance, it's security. If anything, reverse proxying makes things slower (theoretically) because the kernel has to make one extra copy of the data. You reverse proxy stuff so that nginx or haproxy can properly handle I/O management and concurrency, and so they can sanitize HTTP headers, not because it gives you a performance boost. If Go frameworks are multithreaded by default, instead of events, then Go too can benefit from reverse proxying.
If you don't want to think about any kind of reverse proxy setup, there's Passenger Standalone in the Ruby world. You type 'passenger start', and you have a fully-functional, production ready server listening on any port you, that does not require reverse proxying.
As for "fast enough that you don't need to put nginx or haproxy in front of them" - the point of reverse proxying is not performance, it's security. If anything, reverse proxying makes things slower (theoretically) because the kernel has to make one extra copy of the data. You reverse proxy stuff so that nginx or haproxy can properly handle I/O management and concurrency, and so they can sanitize HTTP headers, not because it gives you a performance boost. If Go frameworks are multithreaded by default, instead of events, then Go too can benefit from reverse proxying.
If you don't want to think about any kind of reverse proxy setup, there's Passenger Standalone in the Ruby world. You type 'passenger start', and you have a fully-functional, production ready server listening on any port you, that does not require reverse proxying.
"the point of reverse proxying is not performance, it's security."
Not true. You absolutely need forking/multiple processes on rails deployments to enable even a small number of concurrent requests. If phusion wasn't forking and load balancing between those running instances, your performance would be far lower.
EDIT: See this article from the Passenger folks on tuning a passenger deployment, it should be more clear after reading that how Passenger is dealing with concurrency (by running multiple app instances and load balancing between them): http://blog.phusion.nl/2013/03/12/tuning-phusion-passengers-...
Not true. You absolutely need forking/multiple processes on rails deployments to enable even a small number of concurrent requests. If phusion wasn't forking and load balancing between those running instances, your performance would be far lower.
EDIT: See this article from the Passenger folks on tuning a passenger deployment, it should be more clear after reading that how Passenger is dealing with concurrency (by running multiple app instances and load balancing between them): http://blog.phusion.nl/2013/03/12/tuning-phusion-passengers-...
I wrote that article. :)
The concurrency tuning explained in that article has got nothing to do with reverse proxying. You don't need multiple processes either: the article says that you can use multithreading, and especially in the case of I/O-bound apps, that's enough. If you want to leverage multiple CPU cores however, that's a different story, but it depends on your Ruby implementation. If you use JRuby or Rubinius then threads are enough. But that is still unrelated to the question of whether to reverse proxy or not.
The concurrency tuning explained in that article has got nothing to do with reverse proxying. You don't need multiple processes either: the article says that you can use multithreading, and especially in the case of I/O-bound apps, that's enough. If you want to leverage multiple CPU cores however, that's a different story, but it depends on your Ruby implementation. If you use JRuby or Rubinius then threads are enough. But that is still unrelated to the question of whether to reverse proxy or not.
Don't confuse being able to naively sort of multithread with a runtime that isn't hampered by a Global Interpreter Lock.
That's why there's JRuby and Rubinius.
I agree that the self-contained nature of compiled Go programs is one of Go's most appealing characteristics. In fact, I think you were too diplomatic toward Ruby, and most other popular languages. In my opinion, to insist that distributing a program should not be easy is to expect too little of our tools. I can identify two main deficiencies that make it hard to distribute programs:
1. Lack of a module system that scales well to large programs while still using static linking. According to the talk "Go at Google: Language design in the service of software engineering" (http://talks.golang.org/2012/splash.article), this is one of the main problems that the Go developers aimed to solve.
2. Too much dynamism in the language, such that a packaging tool can't tell with certainty what a program needs, and just as important, what it doesn't need. Go is static enough that this simply doesn't apply. Dart is more static than JavaScript, but still allows some dynamism. The Google Closure tools solve the problem by subsetting JavaScript. Is there any such subset for Python or Ruby? (I don't count RPython, since AFAIK that's only intended for PyPy itself.)
So when choosing a language for a program that one expects to distribute to many users, I think it's a good idea to take these things into account. This may mean choosing a language that lacks some dynamic features that many of us find convenient. (Compile-time metaprogramming can compensate; too bad neither Go nor Dart has it.) Go looks like a pretty good language for servers and command-line utilities.
1. Lack of a module system that scales well to large programs while still using static linking. According to the talk "Go at Google: Language design in the service of software engineering" (http://talks.golang.org/2012/splash.article), this is one of the main problems that the Go developers aimed to solve.
2. Too much dynamism in the language, such that a packaging tool can't tell with certainty what a program needs, and just as important, what it doesn't need. Go is static enough that this simply doesn't apply. Dart is more static than JavaScript, but still allows some dynamism. The Google Closure tools solve the problem by subsetting JavaScript. Is there any such subset for Python or Ruby? (I don't count RPython, since AFAIK that's only intended for PyPy itself.)
So when choosing a language for a program that one expects to distribute to many users, I think it's a good idea to take these things into account. This may mean choosing a language that lacks some dynamic features that many of us find convenient. (Compile-time metaprogramming can compensate; too bad neither Go nor Dart has it.) Go looks like a pretty good language for servers and command-line utilities.
I've written about this before, but this guy definitely wrote it better. I experience a very close relation between this and Ruby vs C#(Mono). I love Ruby as a language, but the pain of using Rubygems and random things requiring Ruby <1.9 and <1.8 (And setting up rvm to manage that) is an absolute pain. Python is slightly better, but I still have a python2 and python3 installed.
This is why I love mono. Things may not be backwards compatible, but I've never had a problem where something wasn't forwards compatible. And I think this is a result of it being compiled. The compilation step decouples the language and syntax from the actual execution of the code. This means that compiled applications can have drastic and breaking syntax improvements, but still be forward compatible on future runtimes. The same can not be said of interpreted languages unless you somehow work a version number into the execution, and even then it makes implementation of the languages much harder because future versions must support every version of the syntax
This is why I love mono. Things may not be backwards compatible, but I've never had a problem where something wasn't forwards compatible. And I think this is a result of it being compiled. The compilation step decouples the language and syntax from the actual execution of the code. This means that compiled applications can have drastic and breaking syntax improvements, but still be forward compatible on future runtimes. The same can not be said of interpreted languages unless you somehow work a version number into the execution, and even then it makes implementation of the languages much harder because future versions must support every version of the syntax
Related to cli and Go, if you ever want to quickly execute Go funcs from a terminal [1]:
$ goe go/parser 'ParseExpr("5 + 7")'
What it needs is a `-e` flag:
ruby -e 'puts 5 + 7'Can you please elaborate on how that would help?
Is it just for familiarity/similarity? I kinda see the 'e' from ruby -e going after the go to form goe. goe -e would be just redundant, wouldn't it?
Is it just for familiarity/similarity? I kinda see the 'e' from ruby -e going after the go to form goe. goe -e would be just redundant, wouldn't it?
Perl, Ruby, Python, and many others have a succinct "evaluate this string" command-line flag. It's so commonplace it's to be expected.
After attempting the distribution of a command line application via Rubygems I came to the following conclusion: Any application not intended exculsively for Rubyists should not be distributed via Rubygems.
Another option could also be to convert gems (and dependency gems) to deb or rpm packages https://github.com/jordansissel/fpm/wiki/ConvertingGems
As far as I know, the Foreman project utilizes this approach http://deb.theforeman.org/pool/precise/stable/f/foreman/
Another option could also be to convert gems (and dependency gems) to deb or rpm packages https://github.com/jordansissel/fpm/wiki/ConvertingGems
As far as I know, the Foreman project utilizes this approach http://deb.theforeman.org/pool/precise/stable/f/foreman/
That is definitely an option, and it is essentially what we did for our linux support when we were building out a cross platform toolchain for mobile games. The tricky part was supporting Windows, Linux, and Mac all at once. Again, not suprised that distribution was hard, there were just a couple things that made distribution in Ruby a more painful experience
In a prior job, I did a decent amount of Python distributions. I had it simple: 1-2 python libraries plugging into a single app which was distributed on a .deb server. Even that was not particularly fun. I can definitely understand the appeal of a build n run system.
As a user, I find rubygem based installs rather horrific (same for pypi & cpan).
n.b.: I'm sure there are tools out there that will appropriately massage your app and put together a app.zip package for you. But IME those sorts of tools are a lot of work to get going.
As a user, I find rubygem based installs rather horrific (same for pypi & cpan).
n.b.: I'm sure there are tools out there that will appropriately massage your app and put together a app.zip package for you. But IME those sorts of tools are a lot of work to get going.
One feature I miss from distributing on RubyGems or PyPI is easy update functionality. Thankfully, go-update[1] looks to solve the problem by making it easy to create self-updating Go binaries.
[1]: https://github.com/inconshreveable/go-update
[1]: https://github.com/inconshreveable/go-update
How many architectures are officially supported by Go?
Straight forward, easy builtin cross compiling is supported from<=>to: darwin/386 darwin/amd64 freebsd/386 freebsd/amd64 linux/386 linux/amd64 linux/arm windows/386 windows/amd64
But, once you start going down the path of other implementations of go (gccgo, gollvm, gollum, goios, etc) the number of platforms supported goes up, but not sure how well supposed / stable the massive list of things go has been ported to is...
But, once you start going down the path of other implementations of go (gccgo, gollvm, gollum, goios, etc) the number of platforms supported goes up, but not sure how well supposed / stable the massive list of things go has been ported to is...
Supported architectures listed here:
http://golang.org/doc/install#requirements
There are two implementations of go: gc and gccgo. The link above describes gc. gccgo is built on gcc and supports many more environments, at least in theory. gccgo will run on addtional processors like powerpc, mips, and sparc and other operating systems like solaris. These aren't necessarily well tested though.
Why don't compile Ruby using f.e. LLVM?
Because it's kinda pointless. The thing that makes Ruby (very) slow isn't the runtime (there might be at most a 1.5x - 2x speedup there), it's the language design. All that dynamism and runtime reflection is really a performance killer.
The dynamism and runtime reflection can be a performance killer. It only is a performance killer in most cases because the Ruby runtimes are still extremely simple as dynamic language implementations go.
Method dispatch in Ruby, for example, can for the typical case (where nobody redefines random methods on all your classes on a regular basis once initial startup has been done) be little more expensive than a C++ virtual member function dispatch, but this requires work to either cache (and invalidate) method lookups, or pre-building vtables (and waste a bit of memory as well as being prepared to propagate updates to them if methods are redefined) for known methods.
Inlining of code is also possible, for example, but requires more work to add "guards" to protect against less common patterns that are still valid.
99% of Ruby is reasonably simple to optimise well. The problem is that the 1% can change the whole world for you and force you to either be able to selectively detect what changes are made or throw out all your optimisations), and so to get fast Ruby implementations will either need to ask for guarantees from the developer (pragma's to promise you'll never alter Fixnum, even in eval() blocks, or flags to by default freeze all standard classes, for example), or they'll need to start employing all the dirty tricks in the book.
Thankfully, the Smalltalk, Lisp, Self and Javascript worlds, amongst others, have pioneered plenty of dirty tricks that'll do just fine (e.g. consider Polymorphic Inline Caches, pioneered in Self, I believe, that are now quite widely applied for dynamic languages)
I think we'll eventually head towards a bit of both: Optional ways of making the job easier, such as promising not to much with things we normally don't muck with anyway, combined with new meta programming libraries to remove the reasons for a lot of the eval() usage, that will allow implementations to make much better judgements about what can actually change (instead of assuming all bets are off after an eval()), and a large number of dirty tricks (in the form of caches + heuristics + guards in various variations, to be able to take fast paths most of the time and "just" fall back to slow code for pathological cases).
Method dispatch in Ruby, for example, can for the typical case (where nobody redefines random methods on all your classes on a regular basis once initial startup has been done) be little more expensive than a C++ virtual member function dispatch, but this requires work to either cache (and invalidate) method lookups, or pre-building vtables (and waste a bit of memory as well as being prepared to propagate updates to them if methods are redefined) for known methods.
Inlining of code is also possible, for example, but requires more work to add "guards" to protect against less common patterns that are still valid.
99% of Ruby is reasonably simple to optimise well. The problem is that the 1% can change the whole world for you and force you to either be able to selectively detect what changes are made or throw out all your optimisations), and so to get fast Ruby implementations will either need to ask for guarantees from the developer (pragma's to promise you'll never alter Fixnum, even in eval() blocks, or flags to by default freeze all standard classes, for example), or they'll need to start employing all the dirty tricks in the book.
Thankfully, the Smalltalk, Lisp, Self and Javascript worlds, amongst others, have pioneered plenty of dirty tricks that'll do just fine (e.g. consider Polymorphic Inline Caches, pioneered in Self, I believe, that are now quite widely applied for dynamic languages)
I think we'll eventually head towards a bit of both: Optional ways of making the job easier, such as promising not to much with things we normally don't muck with anyway, combined with new meta programming libraries to remove the reasons for a lot of the eval() usage, that will allow implementations to make much better judgements about what can actually change (instead of assuming all bets are off after an eval()), and a large number of dirty tricks (in the form of caches + heuristics + guards in various variations, to be able to take fast paths most of the time and "just" fall back to slow code for pathological cases).
The article isn't about performance difference, it's about ease of distribution.
it's the language design. All that dynamism and runtime reflection is really a performance killer.
So why are Lisp and Smalltalk, which are equally dynamic and reflective, even faster?
So why are Lisp and Smalltalk, which are equally dynamic and reflective, even faster?
Ruby does have support for callcc, which most lisps, specially the fast ones like Common Lisp, don't, since it is a pain to implement a fast one without implications in your whole language. Can't comment on smalltalk, since I never programmed in it.
But nobody uses call/cc in Ruby in practice. The issue with Ruby performance is immature VM's/compilers.
There are tons of performance killers in Ruby, but most of them can be "worked around" by making assumptions about the common case and "just" spend some cycles verifying when the assumptions are violated.
E.g. 99.99% of the time, noone are stupid enough to redefine integer addition (because, besides the fact there aren't many good reasons to, all kinds of hell breaks loose if you do, for obvious reasons), so you can usually check the type tag on integers and inline the addition.
The problem is that edge case. But the edge case can be detected when someone tries to redefine '+' and you'd still save massive amounts of time if you were to set a flag at that point and still inline the additions, just with a test and branch to a slow path if people does something stupid (or in the case of JIT's, "just" re-jit everything at that point - yes, then overriding '+' will be slow, but common code will be much faster).
There are tons of performance killers in Ruby, but most of them can be "worked around" by making assumptions about the common case and "just" spend some cycles verifying when the assumptions are violated.
E.g. 99.99% of the time, noone are stupid enough to redefine integer addition (because, besides the fact there aren't many good reasons to, all kinds of hell breaks loose if you do, for obvious reasons), so you can usually check the type tag on integers and inline the addition.
The problem is that edge case. But the edge case can be detected when someone tries to redefine '+' and you'd still save massive amounts of time if you were to set a flag at that point and still inline the additions, just with a test and branch to a slow path if people does something stupid (or in the case of JIT's, "just" re-jit everything at that point - yes, then overriding '+' will be slow, but common code will be much faster).
Ruby does have support for callcc, which most lisps, specially the fast ones like Common Lisp, don't, since it is a pain to implement a fast one without implications in your whole language.
Gambit-C is one of the fastest Lisp implementations around, and it does have a full and performant implementations of call/cc. And I'm afraid that Common Lisp happens to be "fast by sccident" rather than "fast by design" - there's probably more extra speed to be gained by having a better implementable standard (which CL isn't, or at least not that much) than by omitting call/cc.
(Also note that call/cc is sort of out of favour these days, delimited continuations is where it's at.)
Gambit-C is one of the fastest Lisp implementations around, and it does have a full and performant implementations of call/cc. And I'm afraid that Common Lisp happens to be "fast by sccident" rather than "fast by design" - there's probably more extra speed to be gained by having a better implementable standard (which CL isn't, or at least not that much) than by omitting call/cc.
(Also note that call/cc is sort of out of favour these days, delimited continuations is where it's at.)
Well, actually speed was a design goal for Common Lisp. Thus the standard defines the semantics of a compiler, including a file compiler. The CL standard provides stuff like stack allocation, inlining, fast function calling, compiler macros, type system, type declarations, compiler strategy annotations, lots of primitive constructs (like GO in TAGBODY). Most real implementations add several more speed-relevant features.
If you say that Common Lisp 'happens to be fast by accident', then this is completely misleading. 'fast' was a design goal and many implementors have done quite a lot to make demanding applications (like huge object-oriented CAD systems) to be fast.
If you say that Common Lisp 'happens to be fast by accident', then this is completely misleading. 'fast' was a design goal and many implementors have done quite a lot to make demanding applications (like huge object-oriented CAD systems) to be fast.
Well, actually speed was a design goal for Common Lisp.
It might have been one of the design goals, but it wasn't the ultimate design goal (which, I believe, was making a convergent compromise dialect to rule them all), otherwise certain parts of the language would have been most likely simpler. The things you mention are the good stuff. The problem is that there's also some bad stuff that hindered performance of CL on stock hardware until quality compilers were written. Yes, SBCL is quite fast. But you'd have a hard time trying to convince me that even modern implementations of Common Lisp represent some sort of performance optimum in the same way that, say, Stalin Scheme represents an optimum.
Also, wouldn't TAGBODY/GO be replaceable in presence of proper tail recursion by structuring the "TAGBODY basic blocks" into mutually recursive function calls? The same argument could be applied to inlining: Yes, the Common Lisp standard mentions inlining. But did you notice how nowadays, that's more or less considered an implementation issue rather than a language issue? How JVMs are capable of inlining just fine without having to be hinted what to inline and what not to inline?
Now I'd understand some of the hints, perhaps DYNAMIC-EXTENT makes somewhat more sense if the purpose is to say "It's my intent to create this value for a limited period of time, yell at me if I pass it out", but in general, from where I stand, a lot of the design decisions seem to make writing Common Lisp compilers actually more complicated to modern compiler writers.
It might have been one of the design goals, but it wasn't the ultimate design goal (which, I believe, was making a convergent compromise dialect to rule them all), otherwise certain parts of the language would have been most likely simpler. The things you mention are the good stuff. The problem is that there's also some bad stuff that hindered performance of CL on stock hardware until quality compilers were written. Yes, SBCL is quite fast. But you'd have a hard time trying to convince me that even modern implementations of Common Lisp represent some sort of performance optimum in the same way that, say, Stalin Scheme represents an optimum.
Also, wouldn't TAGBODY/GO be replaceable in presence of proper tail recursion by structuring the "TAGBODY basic blocks" into mutually recursive function calls? The same argument could be applied to inlining: Yes, the Common Lisp standard mentions inlining. But did you notice how nowadays, that's more or less considered an implementation issue rather than a language issue? How JVMs are capable of inlining just fine without having to be hinted what to inline and what not to inline?
Now I'd understand some of the hints, perhaps DYNAMIC-EXTENT makes somewhat more sense if the purpose is to say "It's my intent to create this value for a limited period of time, yell at me if I pass it out", but in general, from where I stand, a lot of the design decisions seem to make writing Common Lisp compilers actually more complicated to modern compiler writers.
Common Lisp had several design goals. Speed was an important design goal. The users of Common Lisp wanted to be able to deploy and run large and demanding software on stock hardware.
> ultimate design goal (which, I believe, was making a convergent compromise dialect to rule them all), otherwise certain parts of the language would have been most likely simpler.
No, there was no ultimate design goal and Common Lisp was not designed as a compromise dialect at all. Common Lisp was designed to be a modern replacement for Maclisp and was incorporating several projects working on a modern Maclisp variant, for example NIL, Spicelisp and Zetalisp. Common Lisp was not designed to compromise over Emacs Lisp, Franz Lisp, Portable Standard Lisp, Lelisp, UCI Lisp, Interlisp, Scheme and whatever dialect was there at that time (Common Lisp first round of design was between 1982 and 1984. Then the standardization of ANSI CL was worked on until the early 90s.) If Common Lisp was compromise, then mostly between Maclisp successors. Later during standardization the Condition System of ZetaLisp was integrated, the LOOP of Maclisp and CLOS was developed as a new object system (based on New Flavors from Zetalisp and Portable Common Loops from Xerox). CLOS was also not as a compromise, but a radically new system optionally based on a Meta-Object System.
CLtL1 says: Common Lisp is a new dialect of Lisp, a successor to Maclisp. Influenced strongly by ZetaLisp and to some extent by Scheme and Interlisp.
> that hindered performance of CL on stock hardware until quality compilers were written
Quality compilers were written almost from day one: CMUCL, Allegro CL, LispWorks, Lucid CL. Of those Lucid CL was the fastest with the most optimizations.
Note that there are two or three different types of speed that needs to be addressed in CL: 1) the speed of unoptimized, fully dynamic and safe Lisp. 2) the speed of optimized, non-dynamic, and possibly unsafe Lisp. 3) the speed of production applications which need to be safe, somewhat dynamic, but optimized.
> Also, wouldn't TAGBODY/GO be replaceable in presence of proper tail recursion by structuring the "TAGBODY basic blocks" into mutually recursive function calls?
TCO is done by some implementations. Generally it is not seen as a useful default feature. Common Lisp favors iteration over TCO-based recursion. That's also what I favor. Since there are some implementation targets which don't support TCO (or where TCO makes implementations more difficult, one went to the simpler language). For example some of the Lisp Machines at that time did not support TCO. Today for example the JVM (where Common Lisp also runs on) does not support TCO. Scheme is seen as a different Lisp dialect and CL mostly learned from Scheme the default lexical binding - not more. Remember Scheme is much older than CL. CL did not adopt Scheme's one namespace, its calling semantics and conventions, not its naming conventions, not CALL/CC, not its macro system, ... This again shows that CL is not a compromise dialect - it simply does not support a lot Scheme-style programming - even though Scheme is a decade older than CL. By design.
> Yes, the Common Lisp standard mentions inlining. But did you notice how nowadays, that's more or less considered an implementation issue rather than a language issue? How JVMs are capable of inlining just fine without having to be hinted what to inline and what not to inline?
Common Lisp was not designed for one special implementation platform. Some CL compilers may honor INLINE declarations (many do), others do inlining automatically (like Allegro CL). If an implementation of Common Lisp on the JVM uses its inlining capabilities, fine. But there are many other implementations. ECL for example compiles to C and nobody is would say it is a good thing to stop working on that, just because the JVM exists. The JVM is a language issue. It's the Java Virtual Machine. It was not designed to host Lisp or support efficient implementations of Lisp. One can implement Lisp on the JVM, but it is a not so good fit.
> a lot of the design decisions seem to make writing Common Lisp compilers actually more complicated to modern compiler writers.
You are vague on these issues. Sure a good Common Lisp compiler is complex, but because the complexity is in things like the type system, type inferencing, etc etc. The base language itself is relatively simple. The result is that CL compilers still are producing faster code than most dynamic languages. Native implementations are still much faster and tighter than what the JVM can offer.
> ultimate design goal (which, I believe, was making a convergent compromise dialect to rule them all), otherwise certain parts of the language would have been most likely simpler.
No, there was no ultimate design goal and Common Lisp was not designed as a compromise dialect at all. Common Lisp was designed to be a modern replacement for Maclisp and was incorporating several projects working on a modern Maclisp variant, for example NIL, Spicelisp and Zetalisp. Common Lisp was not designed to compromise over Emacs Lisp, Franz Lisp, Portable Standard Lisp, Lelisp, UCI Lisp, Interlisp, Scheme and whatever dialect was there at that time (Common Lisp first round of design was between 1982 and 1984. Then the standardization of ANSI CL was worked on until the early 90s.) If Common Lisp was compromise, then mostly between Maclisp successors. Later during standardization the Condition System of ZetaLisp was integrated, the LOOP of Maclisp and CLOS was developed as a new object system (based on New Flavors from Zetalisp and Portable Common Loops from Xerox). CLOS was also not as a compromise, but a radically new system optionally based on a Meta-Object System.
CLtL1 says: Common Lisp is a new dialect of Lisp, a successor to Maclisp. Influenced strongly by ZetaLisp and to some extent by Scheme and Interlisp.
> that hindered performance of CL on stock hardware until quality compilers were written
Quality compilers were written almost from day one: CMUCL, Allegro CL, LispWorks, Lucid CL. Of those Lucid CL was the fastest with the most optimizations.
Note that there are two or three different types of speed that needs to be addressed in CL: 1) the speed of unoptimized, fully dynamic and safe Lisp. 2) the speed of optimized, non-dynamic, and possibly unsafe Lisp. 3) the speed of production applications which need to be safe, somewhat dynamic, but optimized.
> Also, wouldn't TAGBODY/GO be replaceable in presence of proper tail recursion by structuring the "TAGBODY basic blocks" into mutually recursive function calls?
TCO is done by some implementations. Generally it is not seen as a useful default feature. Common Lisp favors iteration over TCO-based recursion. That's also what I favor. Since there are some implementation targets which don't support TCO (or where TCO makes implementations more difficult, one went to the simpler language). For example some of the Lisp Machines at that time did not support TCO. Today for example the JVM (where Common Lisp also runs on) does not support TCO. Scheme is seen as a different Lisp dialect and CL mostly learned from Scheme the default lexical binding - not more. Remember Scheme is much older than CL. CL did not adopt Scheme's one namespace, its calling semantics and conventions, not its naming conventions, not CALL/CC, not its macro system, ... This again shows that CL is not a compromise dialect - it simply does not support a lot Scheme-style programming - even though Scheme is a decade older than CL. By design.
> Yes, the Common Lisp standard mentions inlining. But did you notice how nowadays, that's more or less considered an implementation issue rather than a language issue? How JVMs are capable of inlining just fine without having to be hinted what to inline and what not to inline?
Common Lisp was not designed for one special implementation platform. Some CL compilers may honor INLINE declarations (many do), others do inlining automatically (like Allegro CL). If an implementation of Common Lisp on the JVM uses its inlining capabilities, fine. But there are many other implementations. ECL for example compiles to C and nobody is would say it is a good thing to stop working on that, just because the JVM exists. The JVM is a language issue. It's the Java Virtual Machine. It was not designed to host Lisp or support efficient implementations of Lisp. One can implement Lisp on the JVM, but it is a not so good fit.
> a lot of the design decisions seem to make writing Common Lisp compilers actually more complicated to modern compiler writers.
You are vague on these issues. Sure a good Common Lisp compiler is complex, but because the complexity is in things like the type system, type inferencing, etc etc. The base language itself is relatively simple. The result is that CL compilers still are producing faster code than most dynamic languages. Native implementations are still much faster and tighter than what the JVM can offer.
As the Topaz guy said, the slowness is due to the abuse of hash table lookups, allocation and copying (HTLAC) https://speakerdeck.com/alex/why-python-ruby-and-javascript-...
And the abuse of HTLAC patterns is due to the lack of non-HTLAC APIs; the syntax is trascurable
And the abuse of HTLAC patterns is due to the lack of non-HTLAC APIs; the syntax is trascurable
Ruby without dynamism runs a bit faster: http://rubyluwak.com/pages/benchmarks
WTF.... I just looked a bit closer at this, and for starters the graphs are for startup times compared to JRuby, and isn labelled with any values. They say nothing about the performance of the actual code.
Furthermore, it appears to be a commercial product, yet searching for it reveals only a handful of link to their own site, and hardly no mentions elsewhere and the examples/docs are pitiful at best.
Looking quite shady to me.
Furthermore, it appears to be a commercial product, yet searching for it reveals only a handful of link to their own site, and hardly no mentions elsewhere and the examples/docs are pitiful at best.
Looking quite shady to me.
Hi, thanks for your frank comments.
the graphs are for startup times compared to JRuby
The first graph compares startup times. The second graph compares execution times. This has now been clarified on the site.
the examples/docs are pitiful at best
Working on it! What would you like the docs to address specifically?
Looking quite shady to me
I'm sorry you feel that way. Would you like a free evaluation license?
the graphs are for startup times compared to JRuby
The first graph compares startup times. The second graph compares execution times. This has now been clarified on the site.
the examples/docs are pitiful at best
Working on it! What would you like the docs to address specifically?
Looking quite shady to me
I'm sorry you feel that way. Would you like a free evaluation license?
That's interesting, but if you start type annotating the code, then you lose a large part of what makes Ruby attractive to a lot of people.
people write annotations already anyway, in the comments or documentation: see the method signatures at http://ruby-doc.org/core-2.0/Object.html
Yes, take a look at those method signatures. They are imprecise and incomplete. And that is for MRI itself. Very few Ruby projects has documentation anywhere near what MRI does, whether separate or in the comments.
But even if it did: Rubyists are far more likely to be tolerant about documenting type information in optional comments than including them in the code.
I think there's room for experimenting with Ruby's typing, but I also think the most that project can hope to achieve would be extensions that could see some limited use in the odd little piece of performance critical code. It's best hope of success, actually, would be that JRuby and other alternative implementations grow more popular and make it harder for people to rely on C-extensions.
But even if it did: Rubyists are far more likely to be tolerant about documenting type information in optional comments than including them in the code.
I think there's room for experimenting with Ruby's typing, but I also think the most that project can hope to achieve would be extensions that could see some limited use in the odd little piece of performance critical code. It's best hope of success, actually, would be that JRuby and other alternative implementations grow more popular and make it harder for people to rely on C-extensions.
What is the DSL used in the Ruby example?
What's wrong with distributing source + makefile?
I think todays programmers is doing everything back ways. First they think about what language they want to use, then what libraries do they need, and then finally on what platform will it run and how will it be deployed.
In the old days everything was much easier. First you chose a platform, depending on budget, support and so on. When you decided what platform to use, that then dictated what libraries you had at hands, and what package manager you needed for deployment.
That maybe sounds very limiting, but on the other hand you don't always have to come up with new solutions how to bundle and deploy stuff all the time...
In the old days everything was much easier. First you chose a platform, depending on budget, support and so on. When you decided what platform to use, that then dictated what libraries you had at hands, and what package manager you needed for deployment.
That maybe sounds very limiting, but on the other hand you don't always have to come up with new solutions how to bundle and deploy stuff all the time...
Programmers "today"? Maybe I'm not old enough but I cannot remember a time when programmers would choose a platform based on their target userbase, unless they are forced to do so (e.g. iOS). Even back in 2000, given the choice, programmers would always choose to use their favorite language if they can. I remember that back in the days I was very much concerned about distribution. If I write an app in Java, users needed to download a 20 MB runtime (mind you, circa 2000 dialup was common). Visual Basic, Visual C++, etc all required some form of runtime. .NET was in its infancy and not widely distributed. I chose to write as much as possible in Delphi so that users only had 1 executable to worry about, even when Delphi wasn't the most productive platform. I asked questions on various developer forums, and I was being ridiculed for thinking so much about user distribution; the consensus from the community was: pick a language they like, and the user will just have to put up with whatever additional multi-megabyte runtime it requires.
Back in 2000? Good tools like Make where built in the 70s that's utilized by Autotools and other great build systems to bridge the gap between different platforms, so that you don't have to distribute as the author of the article we're discussing, a whole runtime with all dependencies.
But the main point of the article was (which I strongly agree with) that running stuff like Autotools, gems, or package manager kind of sucks when you just want to install a console tool.
At least back in the "old times" we were distributing program + additional runtime, and not requiring a random unexpected runtime in proper version + whole supporting infrastructure + program source code to build. The former option greatly simplified installation.
At least back in the "old times" we were distributing program + additional runtime, and not requiring a random unexpected runtime in proper version + whole supporting infrastructure + program source code to build. The former option greatly simplified installation.
"Since Go is a compiled language, binaries of your app can be precompiled for each platform that you wish to distribute for."
For me, this is worse. There's no audit trail. No way for me to verify that the binary of your app is the same as the source you provide. So there's no way I'm running your binary on my system. I simply cannot trust it with my user account.
Not only is this bad from a security perspective, it's also bad for maintenance. Over time, many end up in a situation where others cannot rebuild binaries from source at all, since binary distribution becomes the norm and the source->binary mechanism isn't maintained except on the developer's own system. Look at the Java community for an example of this. This is a problem because it means that others cannot easy be a member of the community working on the source.
If you want to promote a community around the source (like Github does), you must also provide an easy way to rebuild binaries from the source.
Unfortunately, there's no good answer here.
Distributions like Debian have solved this problem - there are source packages, and there's a standard way of building them to get the binary packages. But there's quite a bit of arcane crud that's built up over the years, making it difficult to learn and understand Debian packaging. And the solution isn't cross platform.
Each language community has produced their own solution (Rubygems, Python eggs, CPAN, etc) but none of them work with the packaging provided by any other language. To me, this may be cross platform, but not being cross language is just as big a problem.