Hi - we think you misunderstood because you probably haven`t red the post properly. WE will implement this data streaming engine, and give it away as open source ! The reason we are asking people if they can solve this challenge, is for us to know if there are other easy to use data streaming engine that can help solve this as per our relaxed/open specifications.
Actually we have been told by another reader recently that Protobuf can in fact encode raw bytes efficiently. We will update the documentation.
Look, why do you even want to use Protobuf? You can just use raw bytes... why even bother with Protobuf - when you can invent everything yourself?
It is pretty clear from the text that Protobuf itself has no way of distinguishing the end of one message and the beginning of the next. Yes, YOU can add that yourself if you want. Like you can add hundreds of other things yourself. But Protobuf doesn't have it.
And - if you really, really love Protobuf so much, let me tell you a secret: You can EASILY embed a Protobuf message inside an IAP message (an ION object field) and there you go... ION works as the message delimiter.
Regarding the "marketing vs. science" comment - We would like YOU to make a comparison of binary protocols that is perfect, solely based on online docs and no access to the core developers... not all docs are equally good, and not everything we have read (e.g.on stackoverflow etc.)seem to be correct.
We don't gain that much by "marketing" a free technology. The customers of VStack.co will be communicating with VStack.co via IAP, regardless of whether IAP catches on outside. We just thought we'd share the work with others.
Yes, we are sure you can build a lot of stuff around Protobuf, but with IAP we will build all that stuff for you.
If that is really true, then we will of course update the comparison page. However, we have put it together from what we were able to find in Google Protocol Buffer's own docs + stack overflow + googling. It is entirely possible that we made mistakes.
Sending schema over the wire is not a good solution for anything else than point-to-point communication. An intermediate node would need every single schema transmitted along with every single messsage, or have another way to keep the schemas cached. That becomes complicated.
The Protobuf documentation says very clearly that you cannot see when one message ends and another begins. Then a protobuf message is not fully self describing. This might be easy to add, but it doesn't have it (according to Protobuf's own docs).
We have looked at Cap'n Proto - but late in the process where we had already looked at quite a lot of formats. From what I can see, Cap'n Proto is pretty much just a binary struct. That is pretty close to what we wanted to do with ION, except we wanted it to be compact on the wire too. We have seen that Cap'n Proto has a compaction mechanism, but we have not yet had time to analyze and compare it to ION's.Cap'n Proto with compaction would be very similar to ION - on a conceptual level.
However, we need to make space for some IAP specific fields coming later in the process (like cache references, column stores and more). Stuff that is IAP specific. That is why we chose to roll with our own encoding in the first place.
Mathias, you also don't need a binary protocol. XML would work. JSON too. But binary is faster and more compact. Same with the table construct. You can work around not having it, but now ION has it built in. You don't make the mistake of serializing an array of JSON objects because you are busy. The objects are serialized as an ION table - not a list of ION object fields. If you ever send an array of objects across the wire with ION (IAP Tools), you will be using the table mode automatically. You save bandwidth and parsing time automatically. Who don't want that - even if you don't need it?
Regarding Copy and Reference, the support for them is still not very good (= not automatic). But imagine your service executes an SQL JOIN query, and in that result a lot of objects are repeated (e.g. same zip + city for a lot of objects). The Copy field can be use to include the zip + city fields just once, and after that refer to them later with a Copy field. That is shorter than including them again. These two fields still need some work to have full support, but we are working on it.
Right now ION is the most well-defined part of IAP. The network protocol itself is still not 100¤ finalized. But, now that we are close to being done with ION (we still have extra fields to add as extended types), we can move forward with the IAP core protocols and semantic protocols. If we do not define a standard semantic protocol for remote object synchronization, IAP will be designed so that you can plug in your own semantic protocol to meet that need.
We believe that while ION is similar in encoding to other formats, it is different in aspects that make ION a better all-round data and network message format. See our ION vs.Other Data Formats Page too.
Not every innovation is from scratch, or 100% unique. MessagePack was designed as a binary encoding of JSON. CBOR too. ION was designed to be a binary general purpose data format able to mimic both JSON, XML, CSV and raw bytes (files), plus the most commonly used data types.
Of course there will be similarities when we try to tackle the same problems, but we also believe we had added something new and useful to the mix.
If you cannot know where one message ends and the next begins without a schema, it is impossible to use Protobuf messages as network messages when messages need to be routed. This should be pretty straightforward logic. Also, they write that Protobuf is not a good strategy for message bigger than 1MB.
From what we have been able to gather from googling about it, it is because Protobuf encodes raw bytes as var-ints which is inefficient. We have not been able to confirm this - Protobuf is weak in this area of the specification.
Not yet. We have been asked to compare ION to Flatbuffers, Cap'n Proto, Thrift, Avro, Transit, BSON and several other encodings. However, writing the benchmarks and going through the features systematically is a lot of work, so we have not yet had the time to go through them all.
No. That is not correct. You cannot route Protobuf messages. You can only do point-to-point communication with Protobuf. This makes Protobuf unsuited for systems that require routing of messages, like a P2P network where not all communication may be point-to-point but have to be routed through intermediate nodes that do not know the schema of the messages sent. Or, systems that communicate via message queues.
Also, since Protobuf is not good at encoding raw bytes (their words, not ours) we find it unsuitable as a general purpose encoding for messages in a network protocol, where we expect a lot of raw bytes to be exchanged (files, video frames, VoIP frames etc.).
Yes, binary formats are not easily readable in a text editor. But, it is actually possible to convert ION to an XML format and back again without loss of information (we have not implemented this yet). This should make it easier to read messages during debug - especially because you don't need to know the schema for the given message to conver it to XML.
Regarding GZipped JSON, it is true that GZiped JSON is small. But, due to the CRIME and BREACH attacks it is not recommended to compress data sent over encrypted connections (TLS).
If you look at our performance benchmarks page you can see a list of serialized length comparisons. As you can see, as soon as you send a few objects in an ION table, the difference is big. More than what you normally can gain with GZip (except perhaps for String).http://tutorials.jenkov.com/iap/ion-performance-benchmarks.h...
Furthermore, GZip only helps with transfer time, and actually slows down parsing time. If you look at our performance benchmarks you will see that ION parsing time is a lot faster than JSON. Additionally, if you really, really want high speed you do not parse ION (or JSON) into Java objects. You process the data directly in its binary form. If you look at our read-and-use benchmark you can see just how big a speed difference that gives. ION is designed for being processed directly. JSON isn't as good for that purpose.
Finally, ION is designed for fast arbitrary hierarchical navigation. JSON is not.
Yes, we have looked at both Apache Avro and Thrift. The ION encoding is similar to these. Avro uses schemas, just like Protobuf, so it is not so easy to route for intermediate nodes. Thrift uses its own IDL schema language too.
We have never claimed that Protobuf could not be used for network applications. We have claimed that Protobuf is a bad choice for messages have have to be routable by intermediaries that do not know the schema of a Protobuf message. Where does one Protobuf message end and another begin?
Additionally, Protobuf is not good at encoding raw bytes - according to their own words.
Yes, we have looked at Transit recently. Transit encodes using JSON or MessagePack. We believe ION to be a more versatile data format than MessagePack, so Transit could benefit from serializing to ION instead of MessagePack.
Yes also, the TLV nature of ION means that you have to buffer the ION data while writing it, in case you don't know the full size of the embedded ahead of time. However, this is really only a problem for very big messages. HTTP has worked like that for a long time already, and HTTP servers have proven capable of scaling pretty well, won't you agree?
Additionally, if a message does not include its length up front, you are just trading faster write time for slower read time. A node receiving dynamically sized data then has the problem of knowing how much data to allocate for the full message, plus the receiver has to inspect the data as it comes in to see where it ends. With an ION message the receiver knows within the first 4-5 bytes (typically) how big that message will be, and can thus copy the following bytes directly into the perfectly allocated memory area, without having to examine them any further. Since data is most often read more times than written (e.g. data written to a file), we felt that making a tradeoff that favors read speed over write speed made sense.
First of all, lots of results sent back from backend services (or databases) are arrays of objects. So no, tables are not a "niche" format - tables are heavily used.
Second, an array of arrays could mean anything. You have not semantics telling whether the arrays are independent or if the first array is an array of columns for the following arrays. ION Tables add that semantic information.
Third, yes, we could encode everything as text or as raw bytes and leave it up to the user to make sense of it. But that is exactly what we are trying to avoid with ION. We want to give devs a decent standard data format to use, that doesn't require a lot of data encoding choices up front. The encoding options have been thought through already, and sensible choices already made which you can just follow.
Fourth, you can nest ION tables inside ION tables, and thus create a more compact representation of an object graph. Using JSON / CBOR you would need a lot of nested arrays inside arrays to emulate that. Possible, but not exactly pretty.
We're working on a certified course that will should be available at https://quantumformalism.academy/courses. It will cover post-quantum cryptography as well.