Google App Engine for Go(code.google.com)
code.google.com
Google App Engine for Go
http://code.google.com/appengine/docs/go/overview.html
15 comments
With the announcement of backends (http://googleappengine.blogspot.com/2011/05/app-engine-150-r...), I suspect long-running processes will be available in Go in the future as well.
Thanks for the info, didn't realize they launched this feature as well. I doubt it will take them long to bring it to Go.
GvR mentioned today at Google I/O that the Go runtime was championed by another group within Google. Apparently, the same thing happened with the Java runtime before it was brought into the App Engine team. He expected the same thing to happen with the Go runtime. So one would expect new features to come to the Go runtime slightly after the other "first class" runtimes (Python and Java).
Why the downvote? This is a legitimate limitation of the App Engine platform, and for Go specifically it eliminates many of the common usecases.
yeah, app engine is pretty limited in that regard. that's why my current go webapp (which needs to send sms and IM) won't run on the app engine. ec2 ahoi ...
This seems like a pretty big step to me: so far Go has seemed like a kind of geeky side project of Google's. But here they are promoting it into one of their production services.
Makes me wonder if they are preparing to slowly edge Java out the door and replace it with something equally performant but not burdened by a hostile owner.
Makes me wonder if they are preparing to slowly edge Java out the door and replace it with something equally performant but not burdened by a hostile owner.
I think you nailed it. Consider they picked up Gosling for r&d I don't put championing their own future platform for all devices (you know Go is going to Android as part of the NDK or something by Android 4).
Isn't this what Sun basically did with Java? I don't know the incumbent they were trying to edge out or if it was just a "look at our cool tech" play.
Go seems like a C-esque version of python/ruby/java to me. A lot of very slick things in there. Lots of room to grow and a lot of community excitement.
Isn't this what Sun basically did with Java? I don't know the incumbent they were trying to edge out or if it was just a "look at our cool tech" play.
Go seems like a C-esque version of python/ruby/java to me. A lot of very slick things in there. Lots of room to grow and a lot of community excitement.
The official blog post: http://blog.golang.org/2011/05/go-and-google-app-engine.html
And Hacker News thread: http://news.ycombinator.com/item?id=2533000
And Hacker News thread: http://news.ycombinator.com/item?id=2533000
According to the blog post, deployment to app engine will only be available initially to those who sign up as a trusted tester via this form: https://spreadsheets.google.com/spreadsheet/viewform?formkey...
Why is the thread [dead]?
Any advantage (performance? memory use?) for using Go vs. Python? Or is just programming language preference?
Faster than Python, better memory use for many types of programs. Compared to Java, less memory use and (I assume) much less start-up time, since no VM is needed. Also, since you can use goroutines, it would seem like you'd be able to do concurrent requests without having to have a special async API like Java/Python have. Go seems like it'll be a pretty great fit for App Engine.
goroutines run in a single OS thread. Same reason for blocking threading on Java. Said this restriction maybe lifted in the future but I wouldn't hold my breath on that.
EDIT: Clarification, on goroutines run in a single OS thread on AppEngine.
EDIT: Clarification, on goroutines run in a single OS thread on AppEngine.
Goroutines still provide concurrency with a single thread. It's not that useful for CPU-bound stuff, but I'd imagine it should allow multiple concurrent GAE API calls (as they are just IO, and certainly async under the hood), or you could do CPU-bound stuff while waiting for IO.
From a google developer, responding to complaints about the new per-instance pricing: "If you are using Java, we just launched the ability to make your instances multi-threaded allowing you to use fewer instances to handle the same
traffic"
https://groups.google.com/group/google-appengine/browse_thre...
I'd be surprised if they didn't do the same with Go, by the time it reaches production status.
I'd be surprised if they didn't do the same with Go, by the time it reaches production status.
Faster than CPython maybe, last time someone did a benchmark of Go vs. Python webservers PyPy came out on top.
I'd be interested in a link to that benchmark if you have it.
I assumed that you're talking about http://ziutek.github.com/web_bench/, but that makes no mention of PyPy (other than a HN comment that PyPy takes 2x the memory) and Go comes out on top in that benchmark.
Either way, when saying Go is faster, I wasn't talking specifically about it's http library or io system (I assume both are similar in speed to Pythons). I more meant that if you're going to be writing custom CPU-intensive code to run on App Engine, assuming the same algorithm, Go should be faster. It has better constant factors, and it gives you better control over allocation.
I don't know PyPy well enough to say in what situations it outperforms Go (especially considering that I don't know what kind of VM warm-up you can expect on App Engine), but since GAE doesn't run PyPy, it's not relevant to the discussion of Python v Go on App Engine.
I assumed that you're talking about http://ziutek.github.com/web_bench/, but that makes no mention of PyPy (other than a HN comment that PyPy takes 2x the memory) and Go comes out on top in that benchmark.
Either way, when saying Go is faster, I wasn't talking specifically about it's http library or io system (I assume both are similar in speed to Pythons). I more meant that if you're going to be writing custom CPU-intensive code to run on App Engine, assuming the same algorithm, Go should be faster. It has better constant factors, and it gives you better control over allocation.
I don't know PyPy well enough to say in what situations it outperforms Go (especially considering that I don't know what kind of VM warm-up you can expect on App Engine), but since GAE doesn't run PyPy, it's not relevant to the discussion of Python v Go on App Engine.
Yep that was the original benchmark, and I ran a PyPy vs. CPython comparison on the same code from which we can extrapolate: http://www.reddit.com/r/Python/comments/fr4w9/benchmarking_g...
And faster doesn't always mean better or more desirable.
I've been enjoying using Go as a C replacement when I need more performance than Python, but don't actually need to use C. It's been fine for that purpose. I would note that they do keep changing the syntax on a fairly regular basis, the libraries are still young, and I do find the language oddly unexpressive in places compared to even C++ with boost. (E.g., no ternary operator, no list comprehensions, using the same word for all loop variants, and so on.) That said, if you're using it as a C replacement, rather than a Python replacement, I think those are fine trade-offs.
Pretty interesting. What do you think of the quote from this article http://news.ycombinator.com/item?id=2518609 http://www.theregister.co.uk/2011/05/05/google_go/print.html
"Google has people who administer apps and services, and they need to, say, write tools that say scrape a few thousand machines statuses and aggregate the data," he says. "Previously, these operations people would write these in Python, but they're finding that Go is much faster in terms of performance and time to actually write the code."
Clever marketing or could Google's problems actually be handled better by Go than Python, in time to code?
"Google has people who administer apps and services, and they need to, say, write tools that say scrape a few thousand machines statuses and aggregate the data," he says. "Previously, these operations people would write these in Python, but they're finding that Go is much faster in terms of performance and time to actually write the code."
Clever marketing or could Google's problems actually be handled better by Go than Python, in time to code?
I think this may be specific to Google's situation--at least for now. PyPI may not be as comprehensive as CPAN, but it's rare these days that I can't "pip install" my way into having half the solution done before I start writing code. The same simply isn't (yet) true for Go.
Google has many custom components, though. I personally don't write Go nearly as quickly as I write Python, but I have 4+ years of writing Python in everything from tiny scripts up to large applications, both for the desktop and for the web. I have maybe twenty to forty hours of experience writing Go code, absolute tops. It's not a fair comparison. If we assume that the Googlers who are claiming to be more productive in Go than Python have a couple orders of magnitude on me, and that they don't need many prefabricated libraries, and I find the claim plausible.
I don't believe it'd currently be true in the general case, though.
Google has many custom components, though. I personally don't write Go nearly as quickly as I write Python, but I have 4+ years of writing Python in everything from tiny scripts up to large applications, both for the desktop and for the web. I have maybe twenty to forty hours of experience writing Go code, absolute tops. It's not a fair comparison. If we assume that the Googlers who are claiming to be more productive in Go than Python have a couple orders of magnitude on me, and that they don't need many prefabricated libraries, and I find the claim plausible.
I don't believe it'd currently be true in the general case, though.
In addition to your custom components point, I wonder if licensing restrictions come into play at all. I am not familiar with Google's policy on use of external code based on license of said code. If there is, hypothetically, a policy to use code for internal projects of only certain licenses, then pypi (generally) may be slightly less useful to them.
If they were re-implementing a significant amount of code then go may indeed be more productive.
If they were re-implementing a significant amount of code then go may indeed be more productive.
They do have a policy regarding external code. It's called "third-party". Google has one big shared code-repo. Generally, the only requirement is to plop the code into the directory called "third-party" and make sure its license is clearly visible. If it's good code, yet "external", Googler's won't have a problem finding a way to use it in there projects.
Have you looked into other Python performance solutions like Cython which lets you add type declarations to Python code, and compile it into a C/C++ module?
In my tests, it produces speedups on numerics that put it within spitting distance of hand-tuned C.
In my tests, it produces speedups on numerics that put it within spitting distance of hand-tuned C.
Oh, no question. That's part of why Mercurial is so close to Git in terms of speed, despite Git being hyper-optimized C and Mercurial being Python. But so far, this has always come up when I'm doing one-off, or nearly one-off, stuff. Getting CPython extensions whipped up is more effort, and less fun, than writing the solution in Go.
None of that is available on the GAE, though (you cannot use custom C extensions on GAE, which includes 3rd party tools like numpy).
My general feeling is that Go is kind of like a lighter Java. Go is compiled, but also optimized for fast compile times, and it should be faster than python. However, from wikipedia: "Of features found in C++ or Java, Go does not include type inheritance, generic programming, assertions, method overloading, or pointer arithmetic."
Ken Thompson is a co-creator, and the language came from Google. I think Google is just showing that they stand behind their language, and it does seem to hit a sweet spot for a lot of web apps. Although, I don't know too many web apps that need better performance than python.
Ken Thompson is a co-creator, and the language came from Google. I think Google is just showing that they stand behind their language, and it does seem to hit a sweet spot for a lot of web apps. Although, I don't know too many web apps that need better performance than python.
"Although, I don't know too many web apps that need better performance than python."
Web apps aren't just CRUD anymore.
http://shootout.alioth.debian.org/u32/benchmark.php?test=all...
If your server in Go is 10x faster than your Python server, which is not necessarily unreasonable, that's 10x less hardware expense you have, if your website is actually "doing something". That can add up.
Web apps aren't just CRUD anymore.
http://shootout.alioth.debian.org/u32/benchmark.php?test=all...
If your server in Go is 10x faster than your Python server, which is not necessarily unreasonable, that's 10x less hardware expense you have, if your website is actually "doing something". That can add up.
> However, from wikipedia: "Of features found in C++ or Java, Go does not include type inheritance, generic programming, assertions, method overloading, or pointer arithmetic."
For many people (including me) the lack of all this "features" is a feature in itself. I certainly have not missed any of them.
For many people (including me) the lack of all this "features" is a feature in itself. I certainly have not missed any of them.
You don't miss generics? In my limited time working with Go, the lack of generics felt like a glaring hole in the language. That said, maybe that feeling goes away as you spend more time with Go, and perhaps learn to program it more idiomatically.
Go has generics, but only for the built-in chan, map, and slice types (and the associated operators). Since the majority of uses of generics (at least in my code) are from containers, this results in me not really feeling the lack too much. Interfaces and reflection cover most of the rest, though not ideally. After using Go for a while, I'm not yet sure if the lack of user-defined generics are a bug or a feature. It limits your expressive power a bit, yes, but it also makes the world so much simpler.
Between interfaces and the builtin generic types, I have never missed generics (specially since append() was added to the language), and that seems to be the experience of most people that have used Go for a while.
Still it probably would be nice to have them some day, but it is worth doing right and not worth sacrificing the current simplicity and elegance of the language.
Still it probably would be nice to have them some day, but it is worth doing right and not worth sacrificing the current simplicity and elegance of the language.
I miss the occasional pointer arithmetic in string manipulations. But that's all I miss of the mentioned features.
For me it's only preference at the moment. My Go code isn't that performance critical that I couldn't implement it with Python. But having not to put up with the whitespace/indentation hell of python is a great relief.
Though the Go ecosystem is nowhere near Python's.
Though the Go ecosystem is nowhere near Python's.
Can anyone confirm if this allows linking native libraries? From the looks of the "compiler", it seems it might be hacked to produce a binary statically linked against, say, SpiderMonkey.
Beyond coolness, it seems a waste of effort to tailor this release to Go, when similar effort might have been expended, e.g. to define a simple protocol talked over a UNIX fd that any static x86 binary could implement to integrate with App Engine.
Beyond coolness, it seems a waste of effort to tailor this release to Go, when similar effort might have been expended, e.g. to define a simple protocol talked over a UNIX fd that any static x86 binary could implement to integrate with App Engine.
There's a lot more than sandboxing to rolling a good App Engine SDK. A large amount of effort went into designing nice, idiomatic Go APIs for accessing the App Engine services.
It looks like the answer is no. You upload your Go source code and App Engine compiles and runs it; you don't upload binaries.
State my assumptions:
GAE allows you to deploy multiple versions of your app and access the same datastore.
Go should significantly out perform Python for certain CPU intensive tasks.
...
If Backends are available at versioned URLs too (and I suspect they do/willdo) then there might be nice opportunities to build little Go Backends to do heavy lifting within your main Python applications. Sounds cool.
...
If Backends are available at versioned URLs too (and I suspect they do/willdo) then there might be nice opportunities to build little Go Backends to do heavy lifting within your main Python applications. Sounds cool.
Go appears to be gain more and more traction. I wonder how long before someone else supports it on their platform.
Great news! I'll start playing with it right away.
I won't drop Python for my day to day coding, but having Go as an option is good to see what cool toys we can build.
I won't drop Python for my day to day coding, but having Go as an option is good to see what cool toys we can build.
Go's a System language. Doesn't that mean it's more suitable for writing a web server, database, driver or OS than as a web site scripting language?
I mean, no one would write System code in PHP, and it's rare to see web sites written in C, so why write a website in Go?
I mean, no one would write System code in PHP, and it's rare to see web sites written in C, so why write a website in Go?
Go turned out to be a pretty nice general purpose language. You got garbage collection, sane native string handling, native maps and lists. Mix that with static typing and you got a pretty nice language that enables fast and sane web development.
Having done a lot of CJK development, I felt that Go's unicode strings were pretty kludgey last time I looked. Go's strings are all utf8, so unless you're working in its ASCII subset alone, you have to manually iterate over multi-byte runes to get the unicode codepoints out. That's really not what I'd call a friendly unicode handling comparable to scripting languages, or even Java.
Please correct if things have changed. I haven't revisited Go for a little while now, and would be very interested as to any updates to its unicode handling.
Please correct if things have changed. I haven't revisited Go for a little while now, and would be very interested as to any updates to its unicode handling.
1. Range on a string iterates over Unicode code points (runes):
While this is all not intuitive (you have to know what does what), I find it rather easy.
s := "Какая-то строка"
for _, rune := range s {
// do something with rune 'К', 'a', ...
}
s := "Какая-то строка"
runes := []int(s)
sub := string(runes[:8]) // "Какая-то"
s[:8] // "Кака"
While this is all not intuitive (you have to know what does what), I find it rather easy.
Iteration over strings is rune-by-rune in Go. However, (somewhat counter-intuitively) string indexing/slicing is byte-by-byte, so you can't just go "str[0:10]" and get the first 10 code points. Then again, that's true in utf16 also, if I'm not mistaken.
But if you want an array of runes instead of a utf8 encoded string, you can just do "[]int(mystring)" and it'll do the conversion for you.
Same in Java, C# and Python. To get the codepoints out or access the nth codepoint you need to iterate over the string if you want your code to be correct and safe.
What's the best way to get started with Google Go?
Start from the left column (Learning Go):
http://golang.org/doc/docs.html and continue to the bottom.
Watch "Practical Go Programming": http://osdc.blip.tv/file/4432146/
Watch "Writing Go Packages": http://www.youtube.com/gocoding#p/u/0/jDWBJOXs_iI
Read language specification.
Edit: If you prefer books, here's a CC-licensed, still in development, book by Miek Gieben "Learning Go": http://www.miek.nl/files/go/ (grab the latest PDF; alternatively, here's Git repo: http://miek.nl/cgi-bin/gitweb.cgi?p=gobook.git;a=summary)
Watch "Practical Go Programming": http://osdc.blip.tv/file/4432146/
Watch "Writing Go Packages": http://www.youtube.com/gocoding#p/u/0/jDWBJOXs_iI
Read language specification.
Edit: If you prefer books, here's a CC-licensed, still in development, book by Miek Gieben "Learning Go": http://www.miek.nl/files/go/ (grab the latest PDF; alternatively, here's Git repo: http://miek.nl/cgi-bin/gitweb.cgi?p=gobook.git;a=summary)
Nice of them to even mark the issue fixed!
http://code.google.com/p/googleappengine/issues/detail?id=23...
http://code.google.com/p/googleappengine/issues/detail?id=23...
[deleted](1)
It's about time! i always wondered why google left out Go! for the app engine last time i was there, since it's their language and stuff. Tough i guess it's ruby next? yeah probably not...
Can't wait to toss Eclipse into the trash!
It's going to be a tough sale for Google to convince people to adopt Go for their web apps when Node.js is taking over that scene. With Node I get the best of two worlds: high concurrency and speed -- presumably the top two selling points for using Go with web apps -- without having to teach myself yet another programming paradigm. Everyone and their mother knows some Javascript; good luck hiring help for your Go-based startup.
IMHO, Google could be riding a much bigger wave right now...
IMHO, Google could be riding a much bigger wave right now...
While I agree that go on appengine might not take off, in no way is node.js "taking over that scene". If by scene you mean the "web app" scene. Just because node.js articles are constantly blowing up hn, reddit does not necessarily correspond to real world deployments of node.js apps.
An admittedly hastily prepared google trends graph: http://www.google.com/trends?q=node.js%2C+python+django%2C+r...
I am a node fan, I subscribe to the node-dev list and have worked on a few applications my self. However I am in no way delusional about the current blog popularity vs actual projects deploying with it.
Also you state that you are able to gain the same concurrency and speed with node that you are with Go. May I ask how you achieve this? Go usually comparable to C code in terms of performance [1]. Go also has amazing concurrency tools for efficient communication between two concurrent goroutines, for example the channel interface[2]. To my knowledge their is no way to run concurrent node processes. You can launch multiple instances of the same program but their is no way to let them communicate without sending data over a socket. You can of course use a reverse proxy to present a single entry point in which multiple node backends can take requests. However you can not claim this as concurrency in the same sense that is available with go.
[1] http://golang.org/doc/go_faq.html#Performance
[2] http://golang.org/doc/effective_go.html#concurrency
An admittedly hastily prepared google trends graph: http://www.google.com/trends?q=node.js%2C+python+django%2C+r...
I am a node fan, I subscribe to the node-dev list and have worked on a few applications my self. However I am in no way delusional about the current blog popularity vs actual projects deploying with it.
Also you state that you are able to gain the same concurrency and speed with node that you are with Go. May I ask how you achieve this? Go usually comparable to C code in terms of performance [1]. Go also has amazing concurrency tools for efficient communication between two concurrent goroutines, for example the channel interface[2]. To my knowledge their is no way to run concurrent node processes. You can launch multiple instances of the same program but their is no way to let them communicate without sending data over a socket. You can of course use a reverse proxy to present a single entry point in which multiple node backends can take requests. However you can not claim this as concurrency in the same sense that is available with go.
[1] http://golang.org/doc/go_faq.html#Performance
[2] http://golang.org/doc/effective_go.html#concurrency
Go looks really nice and I'd really like to learn it especially if I can make a web app or something with it this summer.
Is there something comparable to npm or pip for go? I've seen a few go micro web frameworks but I'd really like to see a list of things like database adapters and the like which have been produced so far. The thing I find amazing about node is just how many great modules have been put out in it's short lifespan (https://github.com/joyent/node/wiki/modules). I suppose Go is a brand new language though.
Thanks!
Is there something comparable to npm or pip for go? I've seen a few go micro web frameworks but I'd really like to see a list of things like database adapters and the like which have been produced so far. The thing I find amazing about node is just how many great modules have been put out in it's short lifespan (https://github.com/joyent/node/wiki/modules). I suppose Go is a brand new language though.
Thanks!
Go has a surprisingly large package list: http://godashboard.appspot.com/project
Like pip or npm, their is goinstall http://golang.org/cmd/goinstall/
Like pip or npm, their is goinstall http://golang.org/cmd/goinstall/
Amazing, thanks!
But with Go can you write your client code and your server code in the same language?
People keep touting this, but the actual number of cases where you really want this is actually pretty limited.
With robust server-side javascript... its not hard to see the application of this. It is easier to generate executable javascript for the client on the server in javascript itself. Certainly in Web2+ you want this for increasingly dynamic behavior?
I do want more dynamism, but I don't see why you should be passing and executing arbitrary code from a source you can only verify with the greatest of difficulty. And having executable code be your communications protocol seems like an absolute nightmare if anything goes wrong. Hell, just versioning it will be hard. It might work for some simple demo apps, but I seriously question the wisdom of such an approach with a team who's size is >1.
Why not pass Json, and write your client code. It's not like Node is so expressive that it's a big win on server side code (especially with the flood of great altjvm & altclr languages these days).
Why not pass Json, and write your client code. It's not like Node is so expressive that it's a big win on server side code (especially with the flood of great altjvm & altclr languages these days).
Maybe some day: http://code.google.com/p/go/issues/detail?id=498
(Native Client is also a possibility.)
(Native Client is also a possibility.)
Sorry, this was actually a tongue-in-cheek comment. :)
I'd like to see more server-side XLST.
> good luck hiring help for your Go-based startup.
If you're unable or unwilling to become fluent in a new language (especially Go, which isn't that difficult to learn), there is no way I'd hire you to program in any language.
Programming ability is multi-dimensional: not everyone is a top-notch as programming language geek AND an algorist AND an architect/designer AND systems hacker. Nonetheless, a programmer should have some aptitude in each of these areas, even if they shine particularly in only one.
A more eloquent summary of this thesis can be found at:
https://twitter.com/#!/moonpolysoft/status/19553333429
If you're unable or unwilling to become fluent in a new language (especially Go, which isn't that difficult to learn), there is no way I'd hire you to program in any language.
Programming ability is multi-dimensional: not everyone is a top-notch as programming language geek AND an algorist AND an architect/designer AND systems hacker. Nonetheless, a programmer should have some aptitude in each of these areas, even if they shine particularly in only one.
A more eloquent summary of this thesis can be found at:
https://twitter.com/#!/moonpolysoft/status/19553333429
With Node I get the best of two worlds: high concurrency and speed
With Node you still need to write in an event-based style, which is something devs often complain about working with Node.js.
"With goroutines and channels, you can say 'I'm going to send a message to another goroutine or wait for a message from another goroutine', but you don't have to actually leave the function you're in to do that," Gerrand says. "That lets you write asynchronous code in a synchronous style. As people, we're much better suited to writing about things in a synchronous style."
from http://www.theregister.co.uk/2011/05/05/google_go/page4.html
With Node you still need to write in an event-based style, which is something devs often complain about working with Node.js.
"With goroutines and channels, you can say 'I'm going to send a message to another goroutine or wait for a message from another goroutine', but you don't have to actually leave the function you're in to do that," Gerrand says. "That lets you write asynchronous code in a synchronous style. As people, we're much better suited to writing about things in a synchronous style."
from http://www.theregister.co.uk/2011/05/05/google_go/page4.html
It's going beyond cute and getting into downright tedious that the Node.js crew think they have some sort of unique lock on concurrency and speed, when instead they are late to the party with an inferior implementation. This meme needs to die. I'd take Go in a heartbeat over Node.js for anything remotely non-trivial.
[deleted]
So you won't be able to run a process like Doozer that communicates with other ports/protocols.
EDIT: More details from the docs at http://code.google.com/appengine/docs/go/runtime.html
An App Engine application cannot:
-write to the filesystem. Applications must use the App Engine datastore for storing persistent data. Reading from the filesystem is allowed, and all application files uploaded with the application are available.
-open a socket or access another host directly. An application can use the App Engine URL fetch service to make HTTP and HTTPS requests to other hosts on ports 80 and 443, respectively.