MessagePack - Extremely efficient object serialization library(msgpack.sourceforge.net)
msgpack.sourceforge.net
MessagePack - Extremely efficient object serialization library
http://msgpack.sourceforge.net/
MessagePack is a binary-based efficient object serialization library. It enables to exchange structured objects between many languages like JSON. But unlike JSON, it is very fast and small.<p>Typical small integer (like flags or error code) is saved only in 1 byte, and typical short string only needs 1 byte except the length of the string itself
20 comments
Why not just use Avro if you want a packed encoding system that's schema-bound, or Fudge if you want one that's schema free and self-describing?
Link to Avro: http://hadoop.apache.org/avro/ Link to Fudge: http://www.fudgemsg.org/ Couldn't do hyperlinks for some reason. Perhaps because I'm a moron.
Link to Avro: http://hadoop.apache.org/avro/ Link to Fudge: http://www.fudgemsg.org/ Couldn't do hyperlinks for some reason. Perhaps because I'm a moron.
I recommend ASN.1 PER over any of the myriad byte-aligned no-schema formats that waste bits on ranges of values and entire types you aren't sending. But this is better than some other formats I've seen.
isn't ASN.1 proprietary? i wanted to implement it once and found i had to pay someone for specs.
is that wrong? if not, i think many people would prefer something more open (more implementations; less likely to die from lack of support)
[edit: sorry, i am talking rubbish here. they are all linked from here, for example: http://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One
i have no idea why i thought they were closed. has the ITU policy changed recently?]
is that wrong? if not, i think many people would prefer something more open (more implementations; less likely to die from lack of support)
[edit: sorry, i am talking rubbish here. they are all linked from here, for example: http://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One
i have no idea why i thought they were closed. has the ITU policy changed recently?]
The specs were definitely closed last time when I checked (about 6 months ago).
Still don't think this old telecom technology is superior compared to modern serialization tools like Protocol Buffers.
Still don't think this old telecom technology is superior compared to modern serialization tools like Protocol Buffers.
old and modern do not mean anything in programming world. C is old - doesn't mean it's better or worse than modern languages. Could you rephrase it in terms of features we can actually compare? I prefer to rely on "old telecom technology" many times, since it's meant to be as close as possible to 100% reliability (telecom being the keyword, not old).
I agree that old and modern are not very appropriate terms when comparing technologies. However I didn't mean to express an objective statement in my comment.
If you send a thousand key/value pairs, protocol buffers waste sixteen thousand bits redundantly expressing the type and tag of each key and value, when the recipient already knows they're all the same. (They revved their format to special-case this but only for a sequence of numbers, not any other type you can construct.)
PER is the only format I've seen that uses the schema to decide how much information the recipient actually needs to decode the message and just sends that, rather than adding pure overhead just in case a recipient has no idea what's going on yet is somehow supposed to do something useful with the message.
PER is the only format I've seen that uses the schema to decide how much information the recipient actually needs to decode the message and just sends that, rather than adding pure overhead just in case a recipient has no idea what's going on yet is somehow supposed to do something useful with the message.
It is true that Protocol Buffers are not optimized for low-mem systems. On the other hand, for preserving bandwidth and storage one can use lightweight compression algorithms on top of general-purpose serialization library. (This is the approach that Google ended up using, AFAIK.)
I think such approach is more flexible than making explicit assumptions on how much data is actually carried in integers.
I think such approach is more flexible than making explicit assumptions on how much data is actually carried in integers.
[deleted]
I bet gzipped JSON compares favorably for larger objects, though, and has the advantage of not being a new protocol. The fact that it supports Haskell and Perl with proper hackagedb / cpan modules is nice, however. Very rare for web 2.0 stuff.
gzipping JSON or XML compares relatively favorably against a pure-binary protocol for suitably large messages, but not for pretty small messages.
In addition, the speed of the compression stream really becomes an issue if you're sending messages within a cluster. Over the internet your bandwidth and latency will start to become more important, but within a data centre all pure-binary protocols are much faster.
In addition, the speed of the compression stream really becomes an issue if you're sending messages within a cluster. Over the internet your bandwidth and latency will start to become more important, but within a data centre all pure-binary protocols are much faster.
I don't understand why JSON would get faster for larger types. Anyway a 524 byte type is already fairly large.
No, the compression algorithms begin to behave as well as hand-optimized binary protocols as the message size increases, because the constant overhead for mapping away the redundancy becomes a small percentage of the overall message.
But if just building the JSON representation already takes more time than the binary protocol, I can't see how adding gzip to it is going to make it faster.
Very interesting, though I'll take more of a look later on. The speed test does seem a little contrived, as it's for one case, and it looks [briefly] like some of the gains are due to greater specificity of primitive types.
Why does their graph show the JSON bar as twice as tall as protocol buffers when it's not even 50% slower.
PB takes 0.47s, JSON takes 1.2s. JSON is 2.55 times slower, so it's over 2 times higher... Not sure where the problem is.
The label for the JSON column on http://msgpack.sourceforge.net/ is "0.68 sec" and the label on the PB column is "0.47sec". So, yes, if the times were what you claimed, the graph would be fine, but according to the graph they were not.
I think we're hitting some browser cache issue. Could you try force-reloading everything? My graph shows the numbers I mentioned above. Maybe the old version showed something different (if I wget the image without any caching, I still get MP: 0.11s, PB: 0.47s, JSON: 1.2s)
Browser cache it is. After a reload the number shows as 1.2.