Gophercloud: A Multi-cloud Software Development Kit for Go(developer.rackspace.com)
developer.rackspace.com
Gophercloud: A Multi-cloud Software Development Kit for Go
http://developer.rackspace.com/blog/gophercloud.html
27 comments
Both Erlang and Go's concurrency models are strongly influenced by Hoare's work in Communicating Sequential Processes. That certainly qualifies. You appear to be interpreting "some attributes" as a deep-dive analysis into the unique traits of what makes Erlang, well, Erlang. That was never the intention, and I'm not sure what motivated this.
Actor model libraries exist in various states of maturity for Go, as well.
Actor model libraries exist in various states of maturity for Go, as well.
Both Erlang and Go's concurrency models are strongly influenced by Hoare's work in Communicating Sequential Processes. That certainly qualifies.
Erlang uses the actor model, while Go uses (a modified version of) CSP. Earlier versions of CSP were more like the actor model in that they did not introduce a level of indirection (the channel). But it's not the same concurrency model.
You appear to be interpreting "some attributes" as a deep-dive analysis into the unique traits of what makes Erlang,
What else am I supposed to do? Otherwise, it would be a statement as meaningless as 'Haskell shares some properties with Algol 68'. I am pretty sure it does.
That was never the intention, and I'm not sure what motivated this.
Erlang is well-known for its concurrency features and the reliability provided by the actor model (including supervision, etc.) (e.g. in telephone systems). It is quite easy to read the quoted sentence as an attempt to give a good reputation to Go by comparing it to Erlang. I only question if such a comparison is warranted.
Actor model libraries exist in various states of maturity for Go, as well.
Well, if they are available in various states of maturity, I am interested to know which one is the most mature. Does it cover most of the functionality of Erlang and Akka?
(The most interesting package that I found some time ago was a binding against the Erlang C nodes functionality.)
Erlang uses the actor model, while Go uses (a modified version of) CSP. Earlier versions of CSP were more like the actor model in that they did not introduce a level of indirection (the channel). But it's not the same concurrency model.
You appear to be interpreting "some attributes" as a deep-dive analysis into the unique traits of what makes Erlang,
What else am I supposed to do? Otherwise, it would be a statement as meaningless as 'Haskell shares some properties with Algol 68'. I am pretty sure it does.
That was never the intention, and I'm not sure what motivated this.
Erlang is well-known for its concurrency features and the reliability provided by the actor model (including supervision, etc.) (e.g. in telephone systems). It is quite easy to read the quoted sentence as an attempt to give a good reputation to Go by comparing it to Erlang. I only question if such a comparison is warranted.
Actor model libraries exist in various states of maturity for Go, as well.
Well, if they are available in various states of maturity, I am interested to know which one is the most mature. Does it cover most of the functionality of Erlang and Akka?
(The most interesting package that I found some time ago was a binding against the Erlang C nodes functionality.)
I think it's a noble effort, but I'd be strongly concerned with the ease of use/digestion/maintainability of something that strives to cover most major cloud APIs. This stuff moves fast, and I can't imagine having to try to keep up with multiple larger platforms.
While we were getting Pathwright launched, we contributed a whole lot to boto, which is primarily (but not exclusively) catered to AWS. It grew Eucalyptus (not a huge jump) and Google Compute support, but they were badly documented/tested and still seem to see much less maintenance than the AWS stuff. The situation has improved slightly as Amazon has hired a few more guys to help the maintainer make boto awesome, but they're still playing catchup with the core competency (AWS). This isn't to talk badly of boto, it's more to illustrate the magnitude of the task of keeping up with even one aggressively developing vendor.
I worry that these "mega-cloud-libs" are the wrong approach. The OpenStack stuff will probably end up with excellent support, but the others are likely to see much less attention, documentation, maintenance, help. The neglected areas of the codebase get to be embarrassing and not up to the quality standards of the primary focus (OpenStack in this case). Perhaps they end up falling behind in API versions for some vendors, limiting their utility.
This is definitely not to poo-poo the effort, I think it's a really neat ideal. Personally, I'd love to see them focus on making a kick ass OpenStack Go API with top notch documentation and tests, and leave the other services to their respective communities. That alone would be noteworthy.
While we were getting Pathwright launched, we contributed a whole lot to boto, which is primarily (but not exclusively) catered to AWS. It grew Eucalyptus (not a huge jump) and Google Compute support, but they were badly documented/tested and still seem to see much less maintenance than the AWS stuff. The situation has improved slightly as Amazon has hired a few more guys to help the maintainer make boto awesome, but they're still playing catchup with the core competency (AWS). This isn't to talk badly of boto, it's more to illustrate the magnitude of the task of keeping up with even one aggressively developing vendor.
I worry that these "mega-cloud-libs" are the wrong approach. The OpenStack stuff will probably end up with excellent support, but the others are likely to see much less attention, documentation, maintenance, help. The neglected areas of the codebase get to be embarrassing and not up to the quality standards of the primary focus (OpenStack in this case). Perhaps they end up falling behind in API versions for some vendors, limiting their utility.
This is definitely not to poo-poo the effort, I think it's a really neat ideal. Personally, I'd love to see them focus on making a kick ass OpenStack Go API with top notch documentation and tests, and leave the other services to their respective communities. That alone would be noteworthy.
Thank you for the feedback - there is definitely a risk of lowest common denominator, and some truth that a generalized toolkit won't be as "good" as a domain-specific tool. For openstack specifically there's goose/juju right now.
We (rackspace) have had great success with Fog, jclouds, pkgcloud, libcloud, and other multi-cloud toolkits. For openstack / Rackspace cloud we also have php-opencloud, openstack.net and pyrax (which uses the python-XXXX clients from openstack core).
It definitely is a fine line to walk; a good abstraction though means that you can use the most common features easily and float through providers, but you have access to the advanced features if you need or want them.
Commonly those advanced features (on a provider level) mean a greater level of vendor lock-in which is why they may not make sense to expose in the higher level abstraction.
We (rackspace) have had great success with Fog, jclouds, pkgcloud, libcloud, and other multi-cloud toolkits. For openstack / Rackspace cloud we also have php-opencloud, openstack.net and pyrax (which uses the python-XXXX clients from openstack core).
It definitely is a fine line to walk; a good abstraction though means that you can use the most common features easily and float through providers, but you have access to the advanced features if you need or want them.
Commonly those advanced features (on a provider level) mean a greater level of vendor lock-in which is why they may not make sense to expose in the higher level abstraction.
I'd also like to point out that having a base-line level of functionality often allows for domain-specific extensions to be written, either as adjunct libraries (c.f., libcloud and rackspace-monitoring libraries for Python), or as integrated extensions in succeeding versions (how Emacs evolved over time).
You're right to think it's impossible to keep up; even getting this far, I've heard rumors and tremors about up-coming V3 APIs in several areas. I'm sure by the time V3 support is contributed, V4 will be on the drawing board. Thankfully, the general availability of a V3 API does not usually preclude the continuing support for V2.
All we can do is deliver value to our users in a consumable format while trying to remain aware of any changes in our ecosystem.
You're right to think it's impossible to keep up; even getting this far, I've heard rumors and tremors about up-coming V3 APIs in several areas. I'm sure by the time V3 support is contributed, V4 will be on the drawing board. Thankfully, the general availability of a V3 API does not usually preclude the continuing support for V2.
All we can do is deliver value to our users in a consumable format while trying to remain aware of any changes in our ecosystem.
First there's a huge caveat to calling this multi-cloud, ie. its a library for openstack clouds only from what i gather of the src and documentation. Second, there's already a nice go library for openstack clouds that's seen multiple cloud usage (rackspace, hpcloud, openstack grizzly/folsom) in the form of http://launchpad.net/goose It has significantly more coverage of available openstack services (image, compute, storage) and more realworld usage and better testing afaics.
No, there is no caveat; we're working on openstack first, then rackspace, then others. It's just started out obviously. We examined goose/juju and preferred to approach this via the Apache 2 license ala all of the other tooling/SDKs out there we (Rackspace) support and assign developers to.
Yes, it is currently focused on openstack clouds, but the goal is to reach parity with Fog (ruby), jclouds (Java), Libcloud (Python), pkgcloud (node.js), and so on.
The focus is a series of clean abstractions for users to be able to build tools and programs which can talk across any cloud provider easily using a common set of names / abstractions.
Yes, it is currently focused on openstack clouds, but the goal is to reach parity with Fog (ruby), jclouds (Java), Libcloud (Python), pkgcloud (node.js), and so on.
The focus is a series of clean abstractions for users to be able to build tools and programs which can talk across any cloud provider easily using a common set of names / abstractions.
sorry.. i couldn't let this go.. how is that not a huge caveat? your advocating and advertising for something that doesn't exist, and that does its existing job worse than other options.. ie a better alternative openstack client lib in go that focuses on openstack. i have huge amount of respect for you from pycon, and maybe this is your first rackspace advocacy... but sigh, all this is a call for developers afaics to work on a vaporware project afaics (going by the standard you set of multicloud). you haven't addressed my example of net security groups and how their elegantly solved by a generic lcd library.. because the fact is rackspace has no net security options in their cloud api, and really its incredibly cloud specific, i'm not talking about passing an a k/v for instance creation but actually configuring security groups (ports, etc), which is pretty basic stuff for a cloud provider.
"I hope users and consumers will keep us honest - as we said in the post, our interest is pretty much no lock-in on an API level (hence: openstack) and no lock-in on the code level."
any api is a lock-in, openstack just happens to be opensource in implementation, but per google v. oracle, there doesn't seem to be any qualms in other opensource impls using the amazon apis (cloudstack, eucalyptus) to deliver what customers want without lock-in.
"I hope users and consumers will keep us honest - as we said in the post, our interest is pretty much no lock-in on an API level (hence: openstack) and no lock-in on the code level."
any api is a lock-in, openstack just happens to be opensource in implementation, but per google v. oracle, there doesn't seem to be any qualms in other opensource impls using the amazon apis (cloudstack, eucalyptus) to deliver what customers want without lock-in.
ic. Probably worth updating the readme then atm its got "Gophercloud provides a Go binding to OpenStack cloud APIs."
fwiw. goose is LGPL.
I'm at a bit of a toss up re LCD between clouds vs exposing the underlying functionality of the clouds. The LCD apis (libcloud from experience) break down with real world usage even for simple things like net security groups, or zones spec into provider specific usage. It seems like most cross cloud apps ends up needing to redefine the abstraction layer within itself rather than attempting to reuse the same LCD code across clouds at which point its not clear that the LCD layer isn't a limitation to taking advantage of a native cloud functionality.
fwiw. goose is LGPL.
I'm at a bit of a toss up re LCD between clouds vs exposing the underlying functionality of the clouds. The LCD apis (libcloud from experience) break down with real world usage even for simple things like net security groups, or zones spec into provider specific usage. It seems like most cross cloud apps ends up needing to redefine the abstraction layer within itself rather than attempting to reuse the same LCD code across clouds at which point its not clear that the LCD layer isn't a limitation to taking advantage of a native cloud functionality.
(disclaimer: I'm one of the Libcloud developers)
I want to add some context to this.
Our current abstraction provides a relatively simple API which is guaranteed to work across multiple providers.
If you want to access provider specific functionality we offer so called extension methods and arguments. This is similar to how jclouds and other cross-provider libraries handle that as well.
We support over 30 providers and for the more popular ones this means you can usually access most of the native cloud functionality through the base API + extension methods.
Doing that will obviously make it harder and more work to switch between different providers so we always put up a disclaimer and make sure our users are aware of that.
We are trying to standardize as many APIs as possible and promote extension methods to the to top level API (proposals and patches are always very welcome and appreciated!), but sadly in a lot of cases this is not possible. Usually this is because the provider APIs are too different and the glue code needed to make it work would be to complex.
In the end it's a trade-off, pretty much like everything else in life and a lot of other things in CS (CAP anyone?).
If you want good portability and easily switch between providers you will stick to the base API. If you don't care so much about the portability you will also use the extension API.
Side note: Even though official provider client libraries exists, a lot of people still tend to use Libcloud with extension methods.
Usually the reason is that we support a variety of Python versions and we have a stricter testing policy which tends to result in less buggy code.
I want to add some context to this.
Our current abstraction provides a relatively simple API which is guaranteed to work across multiple providers.
If you want to access provider specific functionality we offer so called extension methods and arguments. This is similar to how jclouds and other cross-provider libraries handle that as well.
We support over 30 providers and for the more popular ones this means you can usually access most of the native cloud functionality through the base API + extension methods.
Doing that will obviously make it harder and more work to switch between different providers so we always put up a disclaimer and make sure our users are aware of that.
We are trying to standardize as many APIs as possible and promote extension methods to the to top level API (proposals and patches are always very welcome and appreciated!), but sadly in a lot of cases this is not possible. Usually this is because the provider APIs are too different and the glue code needed to make it work would be to complex.
In the end it's a trade-off, pretty much like everything else in life and a lot of other things in CS (CAP anyone?).
If you want good portability and easily switch between providers you will stick to the base API. If you don't care so much about the portability you will also use the extension API.
Side note: Even though official provider client libraries exists, a lot of people still tend to use Libcloud with extension methods.
Usually the reason is that we support a variety of Python versions and we have a stricter testing policy which tends to result in less buggy code.
afaics this only reinforces the use case for just using the underlying native client lib. istm the only portable cloud usage for libcloud is its original use case, read only access for cloudkick monitoring. In some cases its the only decent py lang binding to a given cloud provider and its been great to watch libcloud grow over the years as an apache project that providers target. But the reality check is as an app developer you can get over 90% public cloud marketshare with 3 cloud providers (ec2, openstack, vmwware), and maybe toss in a virt provider masquerading as cloud (ala linode, dotcloud) to get to 95. Given that your extending a from scratch client in libcloud and your abstracting it in an app code base, for most of those you could just go with a native cloud provider and get more functionality. ie. take saltstack's saltcloud as an example of a heavy user of libcloud, they had to create a utility library, and still ended up with 1k lines per cloud of specific code just for something as simple as launching an instance (no volume management, no security group management) just launching an instance. that line count would have dropped and they would have access to more features if they would have just used native cloud libs, if they were willing to grow the per cloud dep. anyways, thats the tradeoff of an lcd library afaics.
Thanks! I'll take care of it or file an issue. As for GPL vs LGPL - I am aware, but given the goals of not focusing on openstack solely and also that all of our other work is Apache 2 for maximum flexibility and freedom, we currently prefer this approach.
The LCD issue is being resolve in libcloud - we have multiple FT devs working on it right now, and does not come up as an issue with Fog, jclouds, or pkgcloud. It's possible to make an abstraction that is simple enough to start but powerful enough to access the advanced features of each individual cloud host.
I hope users and consumers will keep us honest - as we said in the post, our interest is pretty much no lock-in on an API level (hence: openstack) and no lock-in on the code level. We really want to make tools that allow our users and others to fluidly move between hosts.
The LCD issue is being resolve in libcloud - we have multiple FT devs working on it right now, and does not come up as an issue with Fog, jclouds, or pkgcloud. It's possible to make an abstraction that is simple enough to start but powerful enough to access the advanced features of each individual cloud host.
I hope users and consumers will keep us honest - as we said in the post, our interest is pretty much no lock-in on an API level (hence: openstack) and no lock-in on the code level. We really want to make tools that allow our users and others to fluidly move between hosts.
> by Rob Pike and Ken Thompson
Why does everyone always forget Robert Griesemer? He might not be as involved with the public as Rob Pike, but he's the guy that made gofmt and godoc (and I'm pretty sure he's also at least partially to thank for the language being specified in such a way that it is "gofmt"-able in the first place, considering he maintains the spec). That alone counts as a pretty big positive influence on the language.
Why does everyone always forget Robert Griesemer? He might not be as involved with the public as Rob Pike, but he's the guy that made gofmt and godoc (and I'm pretty sure he's also at least partially to thank for the language being specified in such a way that it is "gofmt"-able in the first place, considering he maintains the spec). That alone counts as a pretty big positive influence on the language.
Thanks for that. I'm new to Go myself, and that's some helpful info.
Interesting. Browsing the code it looks like they are using closures[1] in a way I haven't seen other Go code use them. Similar to the "using" keyword in .NET or the "with-" macros (with-open-file) in Common Lisp.
[1] https://github.com/rackspace/gophercloud/blob/master/accepta...
[1] https://github.com/rackspace/gophercloud/blob/master/accepta...
You don't see it because it's not really idiomatic Go code. Another way to do this would be to define a type that is a function type, and then to put a method on that function type. Yes, functions can have methods in Go. This seems a bit odd at first, and it may not seem particularly important, but because a function can have a method, and an interface is just a method set, you can use a function to satisfy an interface. A more idiomatic way to accomplish the same thing would be to define the argument function as it's own type:
shameless plug: I've written about this in more detail here, if this sort of thing interests you: http://jordanorelli.tumblr.com/post/42369331748/function-typ...
type AccessFn func(gophercloud.AccessProvider)
func (fn AccessFn) WithIdentity(allowReauth bool) {
acc, _ := gophercloud.Authenticate(...)
fn(acc)
}
fn := AccessFn(func(gophercloud.AccessProvider ) {
// ...
})
fn.WithIdentity(ar)
shameless plug: I've written about this in more detail here, if this sort of thing interests you: http://jordanorelli.tumblr.com/post/42369331748/function-typ...
Goodness, I'm confused. I'm going to need a lot of time to digest this. That said, much discussion about ideal way of writing Gophercloud happened on the #go-nuts IRC channel, and nobody objected to what I have now as being unidiomatic. You're the first!
Deeply nested closures are definitely a problem, but I usually solve that by factoring closures out into top-level functions of their own.
That said, one of Gophercloud's goals is to be as idiomatic as possible. Are you available on #go-nuts if I have questions? I'd like to play with this technique some more and get a feel for it before applying it to (and risk breaking) gophercloud's existing API.
Deeply nested closures are definitely a problem, but I usually solve that by factoring closures out into top-level functions of their own.
That said, one of Gophercloud's goals is to be as idiomatic as possible. Are you available on #go-nuts if I have questions? I'd like to play with this technique some more and get a feel for it before applying it to (and risk breaking) gophercloud's existing API.
Oh, I'm thick! I see what's happening now!
The problem is it doesn't produce any gain in clarity for the purposes I'm using it for. I think for the acceptance tests, it's use is fine.
I can see if it were part of the public API, there's an argument that an interface could be used, but even there, I'm not sure it'd aid readability.
Semantically, however, they're of course identical.
The problem is it doesn't produce any gain in clarity for the purposes I'm using it for. I think for the acceptance tests, it's use is fine.
I can see if it were part of the public API, there's an argument that an interface could be used, but even there, I'm not sure it'd aid readability.
Semantically, however, they're of course identical.
yeah, to be honest, it may have been eggregious of me to call how I did it idiomatic. There's a lot we just don't have idioms for. Anyway, callback chains tend to be uncommon; that's just one way to avoid it. There's a handful of ways you could write this. Generally if you can expose functionality through methods, things tend to work out better in the end since you can transition into interfaces more readily. I'm jordanorelli in #go-nuts.
I shot this to Samuel the current dev lead - thank you!
Why do I have to read six paragraphs before I find out what Gophercloud is?
That's a lot of handwaving. The overlap is that both Go and Erlang have lightweight threads. But Erlang uses the (fairly) different actor model, which usages messaging between actors, supervision trees to control and monitor the life cycles of actors, remote actors, etc. Go has lightweight threads and channels to communicate between threads. Go (concurrency) shares very little with Erlang.
It's also funny that they move on to claim that Go has better concurrency support than Java, when in fact, you can use real Erlang-like actor networks in Java by adding one dependency to your Maven file. Akka actors are lightweight concurrent entities, since you can easily create hundreds of thousands or even millions of actors.
That said, it's interesting to see one API for multiple clouds.
[1] http://akka.io/