An incomplete guide to Facebook thrift(avabodh.com)

54 pointsby rajeevk12 anni fa20 comments

http://www.avabodh.com/thrift/thrift.html

21 comments

We use Thrift a lot at my company, not just for internal communication, but for our external facing API.

Larger companies like Evernote (https://dev.evernote.com/doc/reference/) have shown that Thrift can be operated successfully at an enormous scale.

Facebook also re-forked re-open-sourced Apache Thrift (originally a Facebook project) as fbthrift: https://github.com/facebook/fbthrift

Xorlev12 anni fa

The unfortunate part of public Thrift APIs is that you must either provide the IDL and hope your consumers are smart enough to work with Thrift, or you have to take the Evernote route and provide SDKs for all the languages you care to support. Admittedly, making SDKs for languages is a lot easier when codegen takes care of 80% of the work.

lmm12 anni fa

Most options for an API offer you the same choice, no? If you're offering XML either you offer a wsdl and hope consumers are smart enough, or you provide SDKs for all the languages. If you're offering JSON you have to provide SDKs for all the languages, since there's not even a standard IDL.

Xorlev12 anni fa

> If you're offering JSON you have to provide SDKs for all the languages, since there's not even a standard IDL.

I disagree. It's best to, but if someone wants to hack together something with python/requests it's not hard to use the returned dictionary. Even with Java, we'll frequently do some requests to get representative payloads, generate models, clean them up and we're on our way. Or you could just use Map<String, Object>.

chuckcode12 anni fa

Does anyone know of a data serialization framework that will do the usual structs and arbitrary size matrices? I'd love to use something like thrift for serializing matrices or other larger data over a wire or onto disk but last time I looked thrift, protocol buffers and avro were all struct oriented and there wasn't a good way to parcel up a matrix or large vector. Currently I've just been using hdf5 as the default way to store reasonably large data matrices, cubes, vectors, etc.

robrenaud12 anni fa

What's wrong with using a repeated float to represent a vector/matrix, perhaps with a repeated int for indices if you are using a sparse representation?

chuckcode12 anni fa

Yes, I could take some time and write it myself. Take a look at hdf5[1] and the features they have around meta information, data chunking, page level compression for efficient reading of sub areas of data spaces. It is a non-trivial amount of functionality but allows some pretty amazing performance, for example see what pytables [2] has built on top of it. Now if they had just made it easier to attach it to a wire as well as a file.

[1] http://www.hdfgroup.org/HDF5/ [2] http://www.pytables.org/moin

jjtheblunt12 anni fa

Lisp (Clojure, ...) reader and pr would do this, no?

Xorlev12 anni fa

I found out the other day that Packetbeat supports decoding Thrift structs:

http://packetbeat.com/docs/configuration.html#thrift-configu...

Which makes debugging Thrift-based architectures significantly easier.

GauntletWizard12 anni fa

This is honestly a better guide to Thrift than we have inside Facebook, though I wish that said more.

rajeevk12 anni fa

Thanks you.. What more you expected in this?

m0skit012 anni fa

Nice introduction. I selected Thrift as middleware for a business project for the company I'm working on one year ago, so I'm pretty familiar with it. It has some issues but for the most part it is reliable, fast, and extremely flexible. Problem with Thrift is mainly the lack of documentation and examples, so the more, the better.

tfigment12 anni fa

I have similar experiences and generally like it. We had to fix one ways in Java at one point (now fixed properly) and not all languages have equal support. We wanted SSL support on a .NET service but nothing official there last I checked. FB updated and standardized SSL on C++ but left out other languages by and large.

Xorlev12 anni fa

And a Scala one (https://github.com/twitter/scrooge, idomatic Scala generated models/parsers) when paired with Finagle/Thrift is pretty great.

electrum12 anni fa

There is a native version of Thrift for Java: https://github.com/facebook/swift

wahsd12 anni fa

"Facebook re-open sourced the thrift..." cool, let me get right on that before it's re-re-closed sourced.

zapf12 anni fa

Good to see this.

I was sold on Thrift but I think the popularity of REST made people take some weird decisions.

Internal services talking to each other using REST seemed like a bit too much, a place, where I though services like Thrift would have worked so well.

Xorlev12 anni fa

I think you mean HTTP+JSON. REST as a philosophy while applied mostly to HTTP isn't really tied to it.

We personally decided to stick with HTTP (even though Thrift would be "superior") since tools are numerous and simple. It's hard to beat curl and a browser. We also found that services end up used in ways that are hard to predict, especially in bash-written tools and the like. Thrift is a barrier to entry.

Additionally, you could do HTTP+Thrift. It's possible, it works, but you lose some of the overhead benefits of Thrift at that point (IMO). In my experience, fulfilling a Thrift request with a Thrift server has tighter latencies at much lower CPU usage for similar request loads.

sbilstein12 anni fa

It's a tough tradeoff between performance flexibility. Relying on HTTP + JSON is very nice when you have lots of teams, codebases in several languages, and want to preserve the ability to talk quickly to any service over HTTP.

I like working HTTP but certainly tools like thrift have there. LinkedIn is all about http://rest.li/ for our services though I've been working with DropWizard on a side project and really love it.

thlt12 anni fa

billowqiu12 anni fa

this is an interested introduction.