Colossus: A New Service Framework from Tumblr(engineering.tumblr.com)
engineering.tumblr.com
Colossus: A New Service Framework from Tumblr
http://engineering.tumblr.com/post/102906359034/colossus-a-new-service-framework-from-tumblr
30 comments
It may not be "directly usable" but I'm sure the code could be examined and patterns gleaned in a way that it contributes something to the greater dialog of code. I know that I spend a lot of time reading different code from all kinds of projects with no intention of actually using the code, just trying to figure out how it's built.
It's both. Internally, the company views it as a recruiting tool. The devs think of it as for the community as well has helping their resumes/cvs.
another possibility: Integration effort at that level requires fairly bright bulbs and bright bulbs rather build than integrate.
> The general structure of a microservice is that it concurrently processes small requests from potentially many clients and keeps little to no internal state.
So what is a microservice exactly? Is it just a piece of code that runs based on a trigger coming from the browser? And what is the main responsibility of the framework? Is it the routing of requests and responses to/from the actual microservices?
Further questions that come to mind: (1) Are the results of microservices cached somehow? (2) Also, if a cached item is not longer required, is it automatically freed? (3) If the output of a running service is no longer required, is that service automatically stopped?
So what is a microservice exactly? Is it just a piece of code that runs based on a trigger coming from the browser? And what is the main responsibility of the framework? Is it the routing of requests and responses to/from the actual microservices?
Further questions that come to mind: (1) Are the results of microservices cached somehow? (2) Also, if a cached item is not longer required, is it automatically freed? (3) If the output of a running service is no longer required, is that service automatically stopped?
In light of your further questions, I feel like I should post.
We're working on a microservice framework which:
... and if this sounds what HTTP servers do, you'd be partially right, but building such apps today is pretty non-trivial and hard to orchestrate.
The focus on microservices is making it much more possible to build apps like this and we're personally betting on an approach which focuses on allowing each microservice to express as much as possible in its own little corner (using our language, which we use both for app structure and communications and gives different programming languages a neutral meeting point).
We're over at http://wym.io/, where there's a demo of our language builder, for now. Hopefully this is low key enough. :P
We're working on a microservice framework which:
- does the routing to and from the actual microservices
- provides a structured enough interface (through our small language), that it can:
- cache the results of requests out to any service
- allow services to find and rely upon each other to compose functionality
- run one lightweight runtime per service per machine (which never shuts down, but has little overhead due to its lack of state)
- serve static assets both to clients and to microservices requesting them
- allow services written in differing languages to talk to each other
... and if this sounds what HTTP servers do, you'd be partially right, but building such apps today is pretty non-trivial and hard to orchestrate.
The focus on microservices is making it much more possible to build apps like this and we're personally betting on an approach which focuses on allowing each microservice to express as much as possible in its own little corner (using our language, which we use both for app structure and communications and gives different programming languages a neutral meeting point).
We're over at http://wym.io/, where there's a demo of our language builder, for now. Hopefully this is low key enough. :P
This sounds very useful.
One further question. Say a service A calls service B. And let's say service B needs several seconds to complete, and emits a progress indicator every second (I assume this is possible?) Will A be able to read these progress indicators also every second, and send its own progress indicator to its caller?
One further question. Say a service A calls service B. And let's say service B needs several seconds to complete, and emits a progress indicator every second (I assume this is possible?) Will A be able to read these progress indicators also every second, and send its own progress indicator to its caller?
It depends on the case, but an easy way to emit progress indicators in our system is to make the long-running service a generator. It looks like this:
A more trivial way is to pass in a callback for the service (we call them routes) to send its updates back to, but this requires more code to send values in.
Edit: to be clear, using a Generator as above isn't very microservice-ey and will trigger our framework to start tracking local state to keep the generator around, which it will auto-free when it believes it can. This is a necessary evil with any long-running code, though. Better is to break up the code so that it may be called multiple times to progress state and returning generator('/someRoute') if possible. :P
g = wym('/email') #=> Generator(id=<...>)
g.on('value', (v) ->
...
)
function next(inp) { ... }
return generator(next);
A more trivial way is to pass in a callback for the service (we call them routes) to send its updates back to, but this requires more code to send values in.
Edit: to be clear, using a Generator as above isn't very microservice-ey and will trigger our framework to start tracking local state to keep the generator around, which it will auto-free when it believes it can. This is a necessary evil with any long-running code, though. Better is to break up the code so that it may be called multiple times to progress state and returning generator('/someRoute') if possible. :P
You'll struggle to find a definition of microservices that everyone will agree on. I'd say microservices are lightweight, isolated units of business logic that can communicate with other such microservices through a well defined interface. I'm sure smarter people than myself can come up with a better definition though.
Well, under your definition, just any piece of code could be a microservice...
So not very enlightening yet :)
So not very enlightening yet :)
Part of the notion is that any piece of code _can_ be a microservice. The way our framework does it, very nearly any function can trivially be turned into a microservice.
It becomes a _service_ once it gains communications and a global name, perhaps?
It becomes a _service_ once it gains communications and a global name, perhaps?
This is the definition most commonly used in the Scala community IME: http://klangism.tumblr.com/post/80087171446/microservices
Whether or not Colossus meets it exactly, I haven't checked.
Whether or not Colossus meets it exactly, I haven't checked.
Why not just use Finagle, which is almost the same thing, NIH aside?
[1] https://twitter.github.io/finagle/
[1] https://twitter.github.io/finagle/
Lead dev here:
Before writing Colossus I had worked with Finagle for several years, and we still use Finagle for some of Tumblr's older systems. There's definitely overlap in functionality between the two and Colossus definitely borrows some ideas from Finagle, but there are also several significant differences:
For the use cases that lead to its development, Colossus is significantly better performing. In the blog post, when I mention "performance problems we faced with existing frameworks we tried", Finagle was the framework we had moved away from. For example, with Colossus you can keep most I/O related code in the event loops without having to use Futures or any other type of out-of-thread concurrency. This alone has a pretty big impact on throughput for the kinds of small requests our services handle. Overall we gained a 4-5x improvement in throughput and much better tail latency.
Finagle still has its own implementations of Future, Try, Duration, and other types that have been part of the Scala standard library for quite some time now. This makes it very unwieldy to use with other popular Scala libraries, especially Akka, which we use quite heavily now. Finagle also still does not build for Scala 2.11 (though it looks like they're close).
Lastly, because Finagle is written on top of Netty, I think a lot of Netty's Java-ness bleeds through: factories, builders, thread pools, etc. Also, between Finagle, its companion util lib, and Netty, you have a massive amount of code to sort through to understand how something works. That becomes a pretty big barrier to entry when trying to do anything beyond the basics, and was a big factor preventing us from writing more services.
Before writing Colossus I had worked with Finagle for several years, and we still use Finagle for some of Tumblr's older systems. There's definitely overlap in functionality between the two and Colossus definitely borrows some ideas from Finagle, but there are also several significant differences:
For the use cases that lead to its development, Colossus is significantly better performing. In the blog post, when I mention "performance problems we faced with existing frameworks we tried", Finagle was the framework we had moved away from. For example, with Colossus you can keep most I/O related code in the event loops without having to use Futures or any other type of out-of-thread concurrency. This alone has a pretty big impact on throughput for the kinds of small requests our services handle. Overall we gained a 4-5x improvement in throughput and much better tail latency.
Finagle still has its own implementations of Future, Try, Duration, and other types that have been part of the Scala standard library for quite some time now. This makes it very unwieldy to use with other popular Scala libraries, especially Akka, which we use quite heavily now. Finagle also still does not build for Scala 2.11 (though it looks like they're close).
Lastly, because Finagle is written on top of Netty, I think a lot of Netty's Java-ness bleeds through: factories, builders, thread pools, etc. Also, between Finagle, its companion util lib, and Netty, you have a massive amount of code to sort through to understand how something works. That becomes a pretty big barrier to entry when trying to do anything beyond the basics, and was a big factor preventing us from writing more services.
Engineer on finagle, used to work at tumblr when we started running into problems with finagle. In large part it was because we didn't understand how it worked, and were doing things like reinventing different pieces of finagle, like load balancing and service discovery.
I think saying colossus is "significantly better performing" is a bit of a stretch, although I haven't seen it in the past year, I think the main win was that you called epoll faster than netty, which I think was because you were using a faster timer. What are the performance comparisons you've done recently?
I think saying colossus is "significantly better performing" is a bit of a stretch, although I haven't seen it in the past year, I think the main win was that you called epoll faster than netty, which I think was because you were using a faster timer. What are the performance comparisons you've done recently?
[deleted]
They are quite similar.
As a longtime Finagle user: the most obvious difference is that Colossus is much more explicit about the event loop, and expects a lot of things to be 'local' to a single event loop. I expect this has nontrivial performance benefits, but it may be a pain when you actually want to share data between threads. (Like, say, a shared cache.)
As a longtime Finagle user: the most obvious difference is that Colossus is much more explicit about the event loop, and expects a lot of things to be 'local' to a single event loop. I expect this has nontrivial performance benefits, but it may be a pain when you actually want to share data between threads. (Like, say, a shared cache.)
> but it may be a pain when you actually want to share data between threads. (Like, say, a shared cache.)
That's the main reason we wanted to make sure it was easy to interface with Akka actors. Assuming you couldn't get away with just using a ConcurrentHashMap or similar, it's very easy to put your shared state into an actor and pass the ActorRef to each event loop, so that all communication between the loops and the actor is through immutable message passing.
Akka's agents (synchronous immediate reads, asynchronous writes) also work really well when you have a read-heavy data-structure like a cache.
That's the main reason we wanted to make sure it was easy to interface with Akka actors. Assuming you couldn't get away with just using a ConcurrentHashMap or similar, it's very easy to put your shared state into an actor and pass the ActorRef to each event loop, so that all communication between the loops and the actor is through immutable message passing.
Akka's agents (synchronous immediate reads, asynchronous writes) also work really well when you have a read-heavy data-structure like a cache.
I'm curious about the motivation to use Scala for such a relatively small codebase.
Presumably, Tumblr also uses Scala for the services themselves... and by assuming Scala, they can make the API more idiomatic.
(For me, the question runs the other way -- I need a very good reason to write a codebase in Java vs. Scala these days, although I understand not everyone feels the same.)
(For me, the question runs the other way -- I need a very good reason to write a codebase in Java vs. Scala these days, although I understand not everyone feels the same.)
A good enough occasion to learn the language?
Not sure if "learning opportunity" should necessarily be on the priority list for a framework that apparently is intended to serve as the backbone for a lot of service development within a company the size of Tumblr ;)
[deleted]
That name is no good. It is reminiscent of “Colossus: The Forbin Project”, a sci-fi apocalypse from 1970 that demonstrates how a single stupid user decision destroys the computing industry for all time. You don't want to remind anyone of that embarrassment.
I've seen that movie 4-5 times. I like it, it's very cheesy.
Every time I watch it, I become more and more convinced Dr Forbin planned it all. :)
Did you know the movie is based on a series of books?
Did you know the movie is based on a series of books?
No, didn't know about the books. I found it on YouTube once and watched it there. Somehow it got stuck in my memory so I was reminded of its title.
[deleted]
It reminds me of the 'Colossus' computer built by Turing, Flowers et. al., and I've never heard of that movie.
It is a slightly odd name for a framework though.
It is a slightly odd name for a framework though.
I see many big products releasing open-source frameworks and most of those frameworks are often very similar to other frameworks that already exists.
Moreover, the goal of those frameworks are also very similar.
But they do it nonetheless. And my understanding for this to happens is that every serious application with enough load is unique in term of architecture and also in terms of culture.
And I believe it's a mix of the two that makes it very hard for any company to use something that is already available. You have the workforce to build something that really looks and talks like you, why would you use something that does only 90% of the job.
And this makes me wonder a lot about why frameworks like that are open-sourced. Is it really for the Greater Good? Or is it a recruiting tool?
I'm ambivalent, and I don't have any answer.