ACK of network transfer is available ("require_ack_response" option). This option ends up choice of at-most-once semantics vs. at-least-once semantics. You need to choose and you can choose.
Fuentd provides "buffer_type file" to buffer records on disk. Shutting down won't loose data. If you need to choose memory buffer for performance reasons, fluentd enables "flush_at_shutdown" option by default.
You would also want to use <secondary> feature. This lets you to write a buffer chunk to another storage if the primary destination is not available "retry_limit" times.
Disclaimer: I'm authoer of MessagePack for C++/Ruby and committer of one for Java.
As for strings, JSON has to allocate memory and copy to deserialize strings because strings are escaped.
MessagePack does't have to allocate/copy because the serialized format of strings is same as the format in memory. But it depends on the implementation whether actually it doesn't allocate/copy.
C++ and Ruby implementations try to suppress allocation and copying (zero-copy). But Java implementation doesn't support zero-copy feature so far (we have plan to do so. Here is "TODO" comment: https://github.com/msgpack/msgpack-java/blob/master/src/main...).
As for the other types, C++ implementation (and new Ruby implementation which is under development) has memory pool which optimizes those memory allocations petterns. But it's hard to implement such optimizations for Java because JVM (and Dalvik VM) doesn't allow to hook object allocation.
We're using MessagePack in a Rails application to log user behaviors and analyze them.
Compared with other serialization libraries such as Protocol Buffers, Avro or BSON, one of the advantages of MessagePack is compatibility with JSON. (In spite of its name, BSON has special types which cause incompatibility with JSON)
It means we can exchange objects sent from browsers (in JSON format) between servers written in different languages without losing information.
I will not use MessagePack with browsers but it's still useful to use it with web applications.
MessagePack includes a concept named "type conversion" to support types which are not supported by its wire format.
With the concept, we can serialize/deserialize user-defined classes as well as strings with encodings.
So far, MessagePack for Java, C++ and D implement the concept.
Although the original blog post focuses on JavaScript and browsers, MessagePack itself doesn't mainly focus on them.
A major use case of MessagePack is to store serialized objects in memcached. A blog post written by Pinterest describes this use case (http://engineering.pinterest.com/posts/2012/memcache-games/).
They use MessagePack with Python which is faster than one with JavaScript. They could store more objects in a server without performance declination (e.g. gzip).
It's true that MessagePack is not always faster than JSON (e.g. within browsers), and it's not always smaller than other serialization methods (e.g. with gzip compression). So we should consider that which serialization methods should I use for "my" case.
There are also general tendency which is helpful to select MessagePack or JSON:
MessagePack is faster to serialize binary data such as thumbnail images.
MessagePack is better to reduce overheads to exchange small objects between servers.
JSON is better to use it with browsers.
We're also using Fluentd as well as original JSON-based logging libraries.
Fluentd deals with JSON-based logs. JSON is good for human facing interface because it is human readable and GREPable.
On the other side, Fluentd handles logs in MessagePack format internally. Msgpack is a serialization format compatible with JSON and can be an efficient replacement of JSON.
I wrote plugin for Fluentd that send those structured logs to Librato Metrics (https://metrics.librato.com/) which provides charting and dashboard features.
With Fluentd, our logs became program-friendly as well as human-firnedly.
Spitting out sprintfs means parsing texts. JSON might be slow since it's text, but binary based formats should be faster especially the log includes many integers.
You can use type-conversion APIs with MessagePack if you use C++, Java or C#. It converts deserialized objects (=dynamically-typed) into statically-typed objects using templates (C++) or reflection + dynamic code generation (Java and C#). You don't have to write boring type-checking codes.
Note that this is a characteristic of the implementation, not of the data format.
Some libraries for JSON like JSONIC (Java) provides these APIs.
Compatibility with JSON is a characteristic of MessagePack.
If you're already using JSON, you can use MessagePack as a faster/smaller substitution of JSON.
Since Protocol Buffers brings their original semantics, you must change lots of codes.
I've used MessagePack on iPhone 3GS for the format of a dictionary file (> 100MB). I utilized mmap and zero-copy deserialization feature of MessagePack.
And I received (and merged) some patches sent by those who use it on iPhone4/iPad2.
I've never tried to run it on MIPS. But it probably works if you use gcc.
- Type mapping between language's type system and serializer's type system
(Note: these serializers are cross-language)
The most understandable difference is "statically typed" vs "dynamically typed". It affects that how to manage compatibility of data and programs.
Statically typed serializers don't store detailed type information of objects into the serialized data, because it is explained in source codes or IDL. Dynamically typed serializers store type information by the side of values.
Generally speaking, statically typed serializers can store objects in fewer bytes. But they they can't detect errors in the IDL (=mismatch of data and IDL). They must believe IDL is correct since data don't include type information. It means statically typed serializers are high-performance but you must strongly care about compatibility of data and programs.
Note that some serializers have original improvements for the problems.
Protocol Buffers store some (not detailed) type information into data. Thus it can detect mismatch of IDL and data. MessagePack stores type information in effective format. Thus its data size becomes smaller than Protocol Buffers or Thrift (depends on data).
Type systems are also important difference. Following list compares type systems of Protocol Buffers, Avro and MessagePack:
Serializers must map these types into/from language's types to achieve cross-language compatibility. It means that some types supported by your favorite language can't be stored by some serializers. Or too many types may cause interoperability problems.
For example, Protocol Buffers doesn't have map (dictionary) type. Avro doesn't tell unsigned integers from signed integers, while Protocol Buffers does. Avro has enum type, while Protocol Buffers and MessagePack don't have.
It was necessary for their designers. Protocol Buffers are initially designed for C++ while Avro for Java. MessagePack aims interoperability with JSON.
I'm using MessagePack to develop our new web service. Dynamically typed and JSON interoperability are required for us.
The article doesn't describe but one of the biggest difference is scalability. MongoDB supports horizontal scaling as a built-in feature. It's known that MySQL can be scaled out by sharding but you lose ACID with it.
Of course scalability is required only when the service goes well. But it's important to considerate it if you intend to be successful.
It can be said scaled MySQL is a no-SQL. MongoDB is the New MySQL with this point of view.
This slide was published by @acmurthy at Hortonworks, which is a new company formed by core Hadoop committers from the Yahoo! Hadoop software engineering team.
http://www.hortonworks.com/
Fuentd provides "buffer_type file" to buffer records on disk. Shutting down won't loose data. If you need to choose memory buffer for performance reasons, fluentd enables "flush_at_shutdown" option by default.
You would also want to use <secondary> feature. This lets you to write a buffer chunk to another storage if the primary destination is not available "retry_limit" times.
Those concerns would be solved by the document: http://docs.fluentd.org/articles/out_forward#buffered-output...