dansimon · HackerTrans

11 ปีที่แล้วdiscuss

Great work! I've been working on a Rust parser combinator library myself ( https://github.com/DanSimon/peruse ), though I've been focusing more on parsing syntax trees than binary formats, and it's really cool to see the different approaches you've taken. I had been planning on working on stream parsers next, but I may just defer to nom since it basically looks like what I had wanted to do.

dansimon

12 ปีที่แล้วdiscuss

> but it may be a pain when you actually want to share data between threads. (Like, say, a shared cache.)

That's the main reason we wanted to make sure it was easy to interface with Akka actors. Assuming you couldn't get away with just using a ConcurrentHashMap or similar, it's very easy to put your shared state into an actor and pass the ActorRef to each event loop, so that all communication between the loops and the actor is through immutable message passing.

Akka's agents (synchronous immediate reads, asynchronous writes) also work really well when you have a read-heavy data-structure like a cache.

dansimon

12 ปีที่แล้วdiscuss

Lead dev here:

Before writing Colossus I had worked with Finagle for several years, and we still use Finagle for some of Tumblr's older systems. There's definitely overlap in functionality between the two and Colossus definitely borrows some ideas from Finagle, but there are also several significant differences:

For the use cases that lead to its development, Colossus is significantly better performing. In the blog post, when I mention "performance problems we faced with existing frameworks we tried", Finagle was the framework we had moved away from. For example, with Colossus you can keep most I/O related code in the event loops without having to use Futures or any other type of out-of-thread concurrency. This alone has a pretty big impact on throughput for the kinds of small requests our services handle. Overall we gained a 4-5x improvement in throughput and much better tail latency.

Finagle still has its own implementations of Future, Try, Duration, and other types that have been part of the Scala standard library for quite some time now. This makes it very unwieldy to use with other popular Scala libraries, especially Akka, which we use quite heavily now. Finagle also still does not build for Scala 2.11 (though it looks like they're close).

Lastly, because Finagle is written on top of Netty, I think a lot of Netty's Java-ness bleeds through: factories, builders, thread pools, etc. Also, between Finagle, its companion util lib, and Netty, you have a massive amount of code to sort through to understand how something works. That becomes a pretty big barrier to entry when trying to do anything beyond the basics, and was a big factor preventing us from writing more services.

comments