XML Sucks(javaoldschool.blogspot.com)
javaoldschool.blogspot.com
XML Sucks
http://javaoldschool.blogspot.com/2009/01/xml-sucks.html
4 comments
"XML is very bad at handling binary data." That is mostly false.
No, it is not. Encode a reasonably sized JPEG (say, 500k-1MB) in an XML document. Try measuring how quickly you can parse that XML document on a lower-end device.
Now try encoding a batch of thumbnails (say, 30k each) in an XML document. Now try seeing how fast you can parse 500 of them on a lower-end device.
There is no obviously winner (widely adopted) standard for binary format XML although there will be some day.
"Some day" doesn't do us any good today.
No, it is not. Encode a reasonably sized JPEG (say, 500k-1MB) in an XML document. Try measuring how quickly you can parse that XML document on a lower-end device.
Now try encoding a batch of thumbnails (say, 30k each) in an XML document. Now try seeing how fast you can parse 500 of them on a lower-end device.
There is no obviously winner (widely adopted) standard for binary format XML although there will be some day.
"Some day" doesn't do us any good today.
XML wasn't really ever made to be high performance though. It was made to be easy to read for humans and relatively easy to generate. At those two things it is awesome. If you are sticking binary data in XML and passing it over the wire to someone you are probably 'doing it wrong'. :)
There are obviously much fast formats for that type of thing, but they have their own drawbacks which are significant. Protocol Buffers are crazy fast but aren't self describing or human readable. JSON is really just a different representation of XML on many levels and certainly doesn't deal with binary data any more elegantly. (base64 here we come!)
If you have binary data and plenty of CPU on both ends, then gzipping XML gets you pretty close to the native size actually and can be a good solution. You keep the human readability and the compression can be done on the transport layer if it is HTTP.
If you don't have much CPU and want easy parsing, it is very easy to write a binary format for XML. I've done this for products that are sending down data to cell phones (including images). The native format is XML, because it is easy to author in templates and easy to debug, but with an extra parameter it gets thrown into a denser and very fast to parse format for the other end to deal with. This costs a bit of CPU on the server, but that generally isn't the problem.
Anyways, I would agree that the main author's points are pretty ridiculous. They either point to him not really understanding XML or never really considering the pros and cons of the alternatives. XML is great for what it is, and it has its drawbacks, but as a universal transport it is pretty rad.
There are obviously much fast formats for that type of thing, but they have their own drawbacks which are significant. Protocol Buffers are crazy fast but aren't self describing or human readable. JSON is really just a different representation of XML on many levels and certainly doesn't deal with binary data any more elegantly. (base64 here we come!)
If you have binary data and plenty of CPU on both ends, then gzipping XML gets you pretty close to the native size actually and can be a good solution. You keep the human readability and the compression can be done on the transport layer if it is HTTP.
If you don't have much CPU and want easy parsing, it is very easy to write a binary format for XML. I've done this for products that are sending down data to cell phones (including images). The native format is XML, because it is easy to author in templates and easy to debug, but with an extra parameter it gets thrown into a denser and very fast to parse format for the other end to deal with. This costs a bit of CPU on the server, but that generally isn't the problem.
Anyways, I would agree that the main author's points are pretty ridiculous. They either point to him not really understanding XML or never really considering the pros and cons of the alternatives. XML is great for what it is, and it has its drawbacks, but as a universal transport it is pretty rad.
There are obviously much fast formats for that type of thing, but they have their own drawbacks which are significant. Protocol Buffers are crazy fast but aren't self describing or human readable.
Why does that matter? The fact that they aren't self-describing more or less requires that you use a published message definition, which solves one of the major problems of XML schemas -- nobody actually uses them, and many that do use them incorrectly.
XML is great for what it is, and it has its drawbacks, but as a universal transport it is pretty rad.
Having wasted far too much of my life on parsing XML documents, I simply can not agree.
XML as a universal transport just sucks compared to the modern alternatives.
Why does that matter? The fact that they aren't self-describing more or less requires that you use a published message definition, which solves one of the major problems of XML schemas -- nobody actually uses them, and many that do use them incorrectly.
XML is great for what it is, and it has its drawbacks, but as a universal transport it is pretty rad.
Having wasted far too much of my life on parsing XML documents, I simply can not agree.
XML as a universal transport just sucks compared to the modern alternatives.
Are you pitting XML against Protocol Buffers? Because they are completely different things. Being self describing is a big deal.. it makes interchange far far easier and obvious. As does being human readable, debugging a web API in a browser or via curl is a serious plus.
Having just spent the past two days reverse engineering an API that used protocol buffers, I can tell you I would have much rather done it against XML. :)
As for parsing, I guess I don't get what the big deal is. I've long since given up on SAX parsers and occasionally use pull parsers, but for the majority of applications using a DOM parser gives you a very OO friendly approach to turning that XML into your objects, and it is dead simple. Sine the generated Protocol Buffer objects are immutable and can't be used as base classes, I suspect the amount of code to turn a 'message' to your domain objects and vice versa is very similar between protocol buffers and a DOM impl. At least it has been in my experience, neither is hard.
Having just spent the past two days reverse engineering an API that used protocol buffers, I can tell you I would have much rather done it against XML. :)
As for parsing, I guess I don't get what the big deal is. I've long since given up on SAX parsers and occasionally use pull parsers, but for the majority of applications using a DOM parser gives you a very OO friendly approach to turning that XML into your objects, and it is dead simple. Sine the generated Protocol Buffer objects are immutable and can't be used as base classes, I suspect the amount of code to turn a 'message' to your domain objects and vice versa is very similar between protocol buffers and a DOM impl. At least it has been in my experience, neither is hard.
Having just spent the past two days reverse engineering an API that used protocol buffers, I can tell you I would have much rather done it against XML.
Why didn't you have the descriptor files? Optimizing for ease of external reverse engineering seems rather counter to the authoring organization's actual goals.
If you want third parties to be able to interoperate, give them the descriptors you use.
As for parsing, I guess I don't get what the big deal is.
Writing lots of code to read and parse data types as text, especially when no schema is available, is time consuming.
Protobuf is necessarily validated. The input types are known-correct, conversion to the domain model is direct and straight-forward, and it's exceptionally easy to generate code or otherwise automatically deserialize to native representations.
Why didn't you have the descriptor files? Optimizing for ease of external reverse engineering seems rather counter to the authoring organization's actual goals.
If you want third parties to be able to interoperate, give them the descriptors you use.
As for parsing, I guess I don't get what the big deal is.
Writing lots of code to read and parse data types as text, especially when no schema is available, is time consuming.
Protobuf is necessarily validated. The input types are known-correct, conversion to the domain model is direct and straight-forward, and it's exceptionally easy to generate code or otherwise automatically deserialize to native representations.
I didn't have the descriptor files because it wasn't an officially supported API. And that happens, lots! Often times people aren't interested in nailing down their API enough to 'officially' publish an interface.. XML makes it easy for you to use it anyways.
Parsing XML doesn't include lots and lots of code, at least not for me. As I said, the amount of code to parse a Proto built class into my own datastructure is almost identical to me doing it from a DOM tree save for some type conversion.
The other plus is that proto buffers involve firing up those tools and incorporating them into your project. How often do you just want something really simple, say the current temp in a zip code? That's a few lines of code to pull out via XML, without the trouble of pulling down the proto buffer definition, building the classes for your language etc..
Parsing XML doesn't include lots and lots of code, at least not for me. As I said, the amount of code to parse a Proto built class into my own datastructure is almost identical to me doing it from a DOM tree save for some type conversion.
The other plus is that proto buffers involve firing up those tools and incorporating them into your project. How often do you just want something really simple, say the current temp in a zip code? That's a few lines of code to pull out via XML, without the trouble of pulling down the proto buffer definition, building the classes for your language etc..
I've never seen value in self-describing or human readable messages. You need some kind of relatively unique identifier for the format of a given message and any optional payload, but beyond that, if a receiver doesn't have any code that knows what to do with a given message, what's the point of being able to deserialize it? And I think debugging an API without making tools is a waste of effort, for the same reason that you use curl rather than typing "POST..." into netcat.
> Protocol Buffers are crazy fast but aren't self describing or human readable.
Protocol Buffers have a text format that is equivalent to the binary format.
Also, XML isn't that much more self-describing than Protocol Buffers. Sure, you can read the tag names in XML, but what is the set of all allowed tag names? What does the stuff inside the tag names actually mean? You have to communicate that stuff somehow, out-of-band, in a spec of some sort. In Protocol Buffer land, a .proto file is just part of that spec.
Protocol Buffers have a text format that is equivalent to the binary format.
Also, XML isn't that much more self-describing than Protocol Buffers. Sure, you can read the tag names in XML, but what is the set of all allowed tag names? What does the stuff inside the tag names actually mean? You have to communicate that stuff somehow, out-of-band, in a spec of some sort. In Protocol Buffer land, a .proto file is just part of that spec.
They have a text format if someone chooses to share it with you. Saying that protocol buffers have a text format makes about as much sense as saying XML has a binary format. Both are true, but it misses the point.
You can get pedantic about needing a spec for everything, but the web isn't really built that way. Do you really track down the full docs for every API you interact with? Say you are just pulling down some information from tumblr via their API or the like? Probably not.. you just throw it in a browser a few times, pull out the fields you need and get on with it. XML lets you do that, quickly and easily and most of the time will work just fine forever more. XML is great for the same reasons HTML is great. It is fast and loose and lets you so easily introspect APIs that you can get your work done quickly.
You can get pedantic about needing a spec for everything, but the web isn't really built that way. Do you really track down the full docs for every API you interact with? Say you are just pulling down some information from tumblr via their API or the like? Probably not.. you just throw it in a browser a few times, pull out the fields you need and get on with it. XML lets you do that, quickly and easily and most of the time will work just fine forever more. XML is great for the same reasons HTML is great. It is fast and loose and lets you so easily introspect APIs that you can get your work done quickly.
There are times when the text format for protocol buffers can be very useful. For example, you could use the text version during development if you wanted human-readability.
If I was pulling some information from Tumblr, I would at least read the introductory API docs at http://www.tumblr.com/docs/api (if you didn't do that, how would you even know what URL to hit?). If they were using Protocol Buffers, they would publish their .proto file on that page. In fact that page reads very much like an informally-specified .proto file; it contains a list of all the parameters and what they all mean.
If I was pulling some information from Tumblr, I would at least read the introductory API docs at http://www.tumblr.com/docs/api (if you didn't do that, how would you even know what URL to hit?). If they were using Protocol Buffers, they would publish their .proto file on that page. In fact that page reads very much like an informally-specified .proto file; it contains a list of all the parameters and what they all mean.
> ... but as a universal transport it is pretty rad
I think that's the key point that everybody misses. XML was really just intended to solve the data interoperability and transport problems, not to function as a datastore. It just so happens that once you've made some XML, or received some from somebody, you may as well just deal with it as is rather than XSLT it into your local apps proprietary format (and as it also turns out, it's fairly easy for people to write by hand and debug in simple cases).
I think that's the key point that everybody misses. XML was really just intended to solve the data interoperability and transport problems, not to function as a datastore. It just so happens that once you've made some XML, or received some from somebody, you may as well just deal with it as is rather than XSLT it into your local apps proprietary format (and as it also turns out, it's fairly easy for people to write by hand and debug in simple cases).
XML is an extensible framework starting with two main things: a data model (very good) and a text-based surface syntax that is portable across a great many character set encodings.
Encoding binary data (as, say, CDATA) and then shipping or storing it in the text-based syntax is every bit the lose you suggest it is. Binary XML formats, however, can answer your objection - the main problem being that no good binary XML format is yet widely adopted.
You say "`Some day' doesn't do us any good today." I half agree. It is true that, today, if you want to use XML for large binary data you have to place your bet on a suitable binary format and you face a rocky road. XML "sucks" in that sense. On the other hand, if you can possibly - for the problem you are trying to solve - make a good bet on a binary format for XML, then it is a fine choice because the data model for XML as interchange format is excellent. So using it for binary today is not an option in many cases and has an element of risk on other cases - but longer term is a winning bet. Your benefit for considering it today is that longer term pay-off. "Go long" on XML.
Encoding binary data (as, say, CDATA) and then shipping or storing it in the text-based syntax is every bit the lose you suggest it is. Binary XML formats, however, can answer your objection - the main problem being that no good binary XML format is yet widely adopted.
You say "`Some day' doesn't do us any good today." I half agree. It is true that, today, if you want to use XML for large binary data you have to place your bet on a suitable binary format and you face a rocky road. XML "sucks" in that sense. On the other hand, if you can possibly - for the problem you are trying to solve - make a good bet on a binary format for XML, then it is a fine choice because the data model for XML as interchange format is excellent. So using it for binary today is not an option in many cases and has an element of risk on other cases - but longer term is a winning bet. Your benefit for considering it today is that longer term pay-off. "Go long" on XML.
XML proponents always trot out "we're just not there yet" or "the basic idea is good, we just need some small fixes", or "parsers just need improvement -- use a better parser library", or any other number of excuses as to why XML doesn't actually suck, it's just a victim of poor implementations or to-be-fixed design inadequacies.
Meanwhile, Protobuf and Thrift work today. They're fast, they're size efficient, and unlike XML, they're simple.
It's time to stop making excuses for XML. It's an inexcusable ball of poorly designed unjustifiable complexity created by architecture astronauts run amok.
I'm disappointed at the upvotes on a whole page of platitudes on why XML isn't a bad format, it's just misunderstood.
Meanwhile, Protobuf and Thrift work today. They're fast, they're size efficient, and unlike XML, they're simple.
It's time to stop making excuses for XML. It's an inexcusable ball of poorly designed unjustifiable complexity created by architecture astronauts run amok.
I'm disappointed at the upvotes on a whole page of platitudes on why XML isn't a bad format, it's just misunderstood.
Got any control characters (below 0x10 iirc) inside your CDATA block? So sorry, expat and conforming parsers will die. How to strip those characters out of your input files in PHP is still one of my most popular blog posts.
XML is horrible for binary data, and the pedantry of parsers makes it worse.
XML is horrible for binary data, and the pedantry of parsers makes it worse.
pedantry-activate: That is a complaint about DTDs, not XML. DTD's are part of the XML specification. While I know what you mean, that DTD's are schema language, and it is natural to think of "XML" as being purely instances within that language, one benefit of including the schema language in the document spec is that some important constraints on the schema language are part of the instance specification (I'm mainly thinking of non-deterministic a.k.a. Unique Particle Attribution a.k.a. 1-unambiguity)
namespaces were "patches pasted on", in logical sense of not being in the xml specification, and in the historical sense of being introduced in afterwards: http://www.w3.org/TR/REC-xml-names/. I think that semantically, they are a wonderful, powerful, useful idea; but syntactically they are awful, confusing, unusable crap.
namespaces were "patches pasted on", in logical sense of not being in the xml specification, and in the historical sense of being introduced in afterwards: http://www.w3.org/TR/REC-xml-names/. I think that semantically, they are a wonderful, powerful, useful idea; but syntactically they are awful, confusing, unusable crap.
"I think that semantically, they are a wonderful, powerful, useful idea;" I agree.
"but syntactically they are awful, confusing, unusable crap." Pretty much the opposite of my experience, and I seem to not be alone. Anecdotally, it's interesting to look at the cat fighting between Dave Winer and the Pubsubhub folks about RSS v. Atom and the constant calls for Winer to sort out his namespace issues. A lot of people seem to find them quite useful and not confusing but, yes, there are people like Winer who (in his case) don't seem to care or (in general) Just Don't Get It. (See my initial comments about the poor state of freely available XML documentation. Someone needs to write the nice 30 page thorough intro to XML concepts and essential details.
"but syntactically they are awful, confusing, unusable crap." Pretty much the opposite of my experience, and I seem to not be alone. Anecdotally, it's interesting to look at the cat fighting between Dave Winer and the Pubsubhub folks about RSS v. Atom and the constant calls for Winer to sort out his namespace issues. A lot of people seem to find them quite useful and not confusing but, yes, there are people like Winer who (in his case) don't seem to care or (in general) Just Don't Get It. (See my initial comments about the poor state of freely available XML documentation. Someone needs to write the nice 30 page thorough intro to XML concepts and essential details.
I probably shouldn't say "crap", it just sounded funny to be so emphatic.
I think, personally, my confusion really comes from xml schema's use of it, in its rules about qualified vs. unqualified etc. http://www.w3.org/TR/xmlschema-0/#NS It's one of those unintuitive sets of rules that I have to look up every time, and it's horrible to get on top of it again.
I think there's probably a de facto standard subset of these rules, that usually used in practice. I bet most visual tools/IDEs make this choice already, so it's invisible to most people who actually work with XML.
I think, personally, my confusion really comes from xml schema's use of it, in its rules about qualified vs. unqualified etc. http://www.w3.org/TR/xmlschema-0/#NS It's one of those unintuitive sets of rules that I have to look up every time, and it's horrible to get on top of it again.
I think there's probably a de facto standard subset of these rules, that usually used in practice. I bet most visual tools/IDEs make this choice already, so it's invisible to most people who actually work with XML.
> As a standard, XML is actually pretty simple and, in my opinion, well thought out.
XML is good at text markup, but not good at data structures, which is what everyone inevitably tries to use it for. This is the basic problem.
XML by itself is a way to create tree structures of text strings. There are no data types (only strings), no schema, and no clear mapping to actual data structures in programming languages.
The evidence of XML's inadequacy for data structure serialization is that it takes a mountain of specifications (XML Schema, DOM) to address all of the needs that XML by itself does not satisfy. And even with all these specifications, there is no simple way to (for example) serialize an array of integers into XML and have them get deserialized into the same ten integers, without doing your own text manipulation.
XML has had ten years of almost no serious competition and the favor of the w3c and the web community, and all we have to show for it is an ever growing pile of specs that still fail to make data interchange easy. Also, XML is horribly inefficient.
What is the right answer? Protocol Buffers. They are 3-10x smaller, 20-100x faster to parse, they have a schema built in with real data types at the lowest level of the stack, they have both a binary and a text format, and they have a much more natural mapping onto real data structures. Compare:
XML is good at text markup, but not good at data structures, which is what everyone inevitably tries to use it for. This is the basic problem.
XML by itself is a way to create tree structures of text strings. There are no data types (only strings), no schema, and no clear mapping to actual data structures in programming languages.
The evidence of XML's inadequacy for data structure serialization is that it takes a mountain of specifications (XML Schema, DOM) to address all of the needs that XML by itself does not satisfy. And even with all these specifications, there is no simple way to (for example) serialize an array of integers into XML and have them get deserialized into the same ten integers, without doing your own text manipulation.
XML has had ten years of almost no serious competition and the favor of the w3c and the web community, and all we have to show for it is an ever growing pile of specs that still fail to make data interchange easy. Also, XML is horribly inefficient.
What is the right answer? Protocol Buffers. They are 3-10x smaller, 20-100x faster to parse, they have a schema built in with real data types at the lowest level of the stack, they have both a binary and a text format, and they have a much more natural mapping onto real data structures. Compare:
cout << "Name: " << person.name() << endl;
cout << "E-mail: " << person.email() << endl;
cout << "Name: "
<< person.getElementsByTagName("name")->item(0)->innerText()
<< endl;
cout << "E-mail: "
<< person.getElementsByTagName("email")->item(0)->innerText()
<< endl;
The real problem, and I use this article and your post as evidence, is that after more than a decade, XML is poorly documented, poorly understood and hard to use. The tools stink, and unspecified proscriptions to "use good tools" pervade the XML community yet nobody seems to really be writing "good tools" and there seems to be very little consensus about what is a "good tool".
What I mean by poorly documented is this, to people unfamiliar with XML, they head over to their local bookstore to pick up a guide or somesuch and find a 2000 page tome with 20 authors on the cover setting forward a several month path of reading and writing code to understand the concepts in the book, and probably finding little illumination on the topic to help them grok the part of the vast XML spec they need to parse and map relatively simple data structures to their software -- something that should take a half page tutorial in some random XML library's documentation.
And I agree, DTDs and XSD are in a sense external to XML, but when every 2000 page tome you pick up hammers away about XML validity, you feel like you have to support those too -- and supporting those adds vast amounts of complexity to your application, the XML format you are using, the message interchange, etc. (so nobody who knows anything uses them anyways).
So what should be a 2 hour development task turns into a 3 month hair pulling and library testing exercise.
What I mean by poorly documented is this, to people unfamiliar with XML, they head over to their local bookstore to pick up a guide or somesuch and find a 2000 page tome with 20 authors on the cover setting forward a several month path of reading and writing code to understand the concepts in the book, and probably finding little illumination on the topic to help them grok the part of the vast XML spec they need to parse and map relatively simple data structures to their software -- something that should take a half page tutorial in some random XML library's documentation.
And I agree, DTDs and XSD are in a sense external to XML, but when every 2000 page tome you pick up hammers away about XML validity, you feel like you have to support those too -- and supporting those adds vast amounts of complexity to your application, the XML format you are using, the message interchange, etc. (so nobody who knows anything uses them anyways).
So what should be a 2 hour development task turns into a 3 month hair pulling and library testing exercise.
I'm trying to figure out whether this comment is serious, or a quite funny joke about xml being verbose to the point that human readability is sacrificed.
It's serious.
In small snippets, the end tags and "<...>" nonsense actually aids readability, I think. Not that I've done any experiments to test that.
In large snippets, those things are a pain for both readability and writability.
However, there's a catch: the difference between a well written plain text editor and one that gives you a nice "structure editor" for XML is fairly small. That is, a good editor that spares you typing so much to author XML and that makes XML easier to read -- not so much more complicated than a regular old plain text editor.
So, to some extent, I really don't care about the problems of the surface syntax of XML for large snippets. I feel the same way about S-exps in lisp. They too are just fine in small snippets and, for large datums - use a structure editor.
In small snippets, the end tags and "<...>" nonsense actually aids readability, I think. Not that I've done any experiments to test that.
In large snippets, those things are a pain for both readability and writability.
However, there's a catch: the difference between a well written plain text editor and one that gives you a nice "structure editor" for XML is fairly small. That is, a good editor that spares you typing so much to author XML and that makes XML easier to read -- not so much more complicated than a regular old plain text editor.
So, to some extent, I really don't care about the problems of the surface syntax of XML for large snippets. I feel the same way about S-exps in lisp. They too are just fine in small snippets and, for large datums - use a structure editor.
In small snippets, the end tags and "<...>" nonsense actually aids readability, I think.
Baaaaaaaaaaaaaaaaawhahahahahahahahahahahah oh man
Feel free to downvote me, I admit I don't have the cognitive ability to offer an compelling response to that after laughing so hard.
Baaaaaaaaaaaaaaaaawhahahahahahahahahahahah oh man
Feel free to downvote me, I admit I don't have the cognitive ability to offer an compelling response to that after laughing so hard.
There's one case where I like XML. Interoperating with someone who, starting with a clean sheet, would have concocted something worse than XML (EDI is a popular example).
That said, I really don't see the value of whitespace and comments in our RPC traffic. I wish the industry were mature enough to successfully use ASN.1 for this stuff.
That said, I really don't see the value of whitespace and comments in our RPC traffic. I wish the industry were mature enough to successfully use ASN.1 for this stuff.
ASN.1 is no better. You get some space efficiency (and maybe some CPU efficiency depending on how special-cased your parser is) at the cost of a specification that is, frankly, much more complicated than XML.
The use of text is the right thing; if you really care about performance at a level that naively compressed text formats can't provide, you frankly need to be defining your own wire protocols anyway.
The use of text is the right thing; if you really care about performance at a level that naively compressed text formats can't provide, you frankly need to be defining your own wire protocols anyway.
The use of text is the right thing;
Why? It's genuinely inefficient. This is not academic, as easily illustrated by serializing even a megabyte of binary data in a text-encoded document. Base64+GZIP is not a solution.
... if you really care about performance at a level that naively compressed text formats can't provide ...
Why are we optimizing a transfer serialization for your own readability and not the devices we're actually targeting?
I've wasted days trying to get barely acceptable XML parsing performance on the iPhone, and similar lower-end ARM devices are becoming much more prevalent these days.
... you frankly need to be defining your own wire protocols anyway.
Protocols like XMPP dictate the use of XML. There wouldn't be nearly as many performance issues if they'd used something akin to Thrift, Protobuf, or even the woefully misunderstood ASN.1 DER.
Since XMPP uses XML, they've had to define complex out-of-band protocols for transfering file data, instead of simply sending the data in-line as simple binary XMPP messages.
I could not more strongly disagree with your position. Protocol implementations should use well-defined, efficient binary serialization mechanisms (such as Protobuf or Thrift) wherever possible to maximize performance, and minimize both serialization overhead and implementation complexity.
Why? It's genuinely inefficient. This is not academic, as easily illustrated by serializing even a megabyte of binary data in a text-encoded document. Base64+GZIP is not a solution.
... if you really care about performance at a level that naively compressed text formats can't provide ...
Why are we optimizing a transfer serialization for your own readability and not the devices we're actually targeting?
I've wasted days trying to get barely acceptable XML parsing performance on the iPhone, and similar lower-end ARM devices are becoming much more prevalent these days.
... you frankly need to be defining your own wire protocols anyway.
Protocols like XMPP dictate the use of XML. There wouldn't be nearly as many performance issues if they'd used something akin to Thrift, Protobuf, or even the woefully misunderstood ASN.1 DER.
Since XMPP uses XML, they've had to define complex out-of-band protocols for transfering file data, instead of simply sending the data in-line as simple binary XMPP messages.
I could not more strongly disagree with your position. Protocol implementations should use well-defined, efficient binary serialization mechanisms (such as Protobuf or Thrift) wherever possible to maximize performance, and minimize both serialization overhead and implementation complexity.
Even with a binary format, you'd be sending data out of band in XMPP. XMPP is designed for interchanging _small_ bits of data. Large data blocks destroy latency, so they are usually sent out of band.
Plenty of large data blocks occur without any binary file transfers, like disco items responses from large services.
The solution to this problem isn't a binary encoding of XMPP, it's multiple simultaneous streams. The problem exists regardless of the encoding. Some of the solutions for this set up internal streams (In-band bytestreams and in-band Jingle are two examples). Since well defined protocols already exist for voice/video and other such things, those are used when possible. This isn't a bad thing. No technology needs to do everything itself.
Also, the complexity of out-of-band protocols in XMPP has nothing at all to do with XML. It has to do with NATs and firewalls. It is not easy to set up peer-to-peer communication on the Internet that actually works. SIP and RTP and their ilk are quite complex, I agree. Jingle also has many simple out-of-band and in-band set ups you can use as well, but they aren't nearly as complete or flexible.
Plenty of large data blocks occur without any binary file transfers, like disco items responses from large services.
The solution to this problem isn't a binary encoding of XMPP, it's multiple simultaneous streams. The problem exists regardless of the encoding. Some of the solutions for this set up internal streams (In-band bytestreams and in-band Jingle are two examples). Since well defined protocols already exist for voice/video and other such things, those are used when possible. This isn't a bad thing. No technology needs to do everything itself.
Also, the complexity of out-of-band protocols in XMPP has nothing at all to do with XML. It has to do with NATs and firewalls. It is not easy to set up peer-to-peer communication on the Internet that actually works. SIP and RTP and their ilk are quite complex, I agree. Jingle also has many simple out-of-band and in-band set ups you can use as well, but they aren't nearly as complete or flexible.
Even with a binary format, you'd be sending data out of band in XMPP. XMPP is designed for interchanging _small_ bits of data.
Having implemented both XMPP clients and servers, I don't particularly see your point. Exchanging small binary message packets would not be an issue.
How would binary stanzas be any different than other message-based multiplexing binary protocols?.
Also, the complexity of out-of-band protocols in XMPP has nothing at all to do with XML.
The fact that you have to go OOB at all introduces significant complexity.
Having implemented both XMPP clients and servers, I don't particularly see your point. Exchanging small binary message packets would not be an issue.
How would binary stanzas be any different than other message-based multiplexing binary protocols?.
Also, the complexity of out-of-band protocols in XMPP has nothing at all to do with XML.
The fact that you have to go OOB at all introduces significant complexity.
There must be some campaign of secrecy around ASN.1. When I mention it to otherwise well-informed developers, they have never heard of it. It doesn't help that the really high performance marshalling libraries aren't open source.
There was a real time blog search engine, pubsub.com. The internal messages was encoded in ASN.1. Bob Wyman, CTO of the pubsub.com was part of the ASN.1 committees in 80's.
How can no one hear ASN.1? Andy Tanenbaum's Computer Network mentions it. SSL uses part of it.
How can no one hear ASN.1? Andy Tanenbaum's Computer Network mentions it. SSL uses part of it.
Eugene, I wasn't actually a formal member of the standards committee, however, since my products at DEC (ALL-IN-1 mail, etc.) were some of the earliest to use ASN.1, I was very involved in the standardization process.
ASN.1 is everywhere whether or not people realize it. Just about all telecommunications systems (cell phones, etc.) use it internally. As you say, SSL uses it -- since public key certificates are encoded in a form of ASN.1 also the IETF SNMP protocol relies on ASN.1. Other examples would include X.400 and X.500 standards and the LDAP which is a simplified version of X.500.
You may be interested to know that internally at Google, we use the "Protocol Buffer" format instead of XML for all internal processing. That format is, like ASN.1 a "tag/length/value" binary encoding. For more information see: http://code.google.com/apis/protocolbuffers/
Bob, I know that. and Facebook uses Thrift which is now part of Apache incubator project. I may need to deal with some large data set with Cassandra database, which communicates with external programs in Thrift format messages.
As a side note, I hope Google will finish real time search api soon and open the firehose :-)
As a side note, I hope Google will finish real time search api soon and open the firehose :-)
I've heard of it, but have no direct experience. Everyone I've met who has direct experience just shudders whenever it is mentioned. My understanding is that it is extremely complex and difficult to implement.
The primary issues with ASN.1 seem to be that the availability libraries are poor and the documentation is both hard to find, and difficult to follow.
> Very bad at handling binary data
Agreed. This is common to most text formats though. It This is why most XML apps use URL relationships to refer to binaries.
> Incredibly verbose
There's a great deal on confusion between XML the data structure and XML the common method of serializing that data structure. Probably because they have the same name.
Primarily, we deal with XML the data structure. If we create something from scratch, or once we de-serialize what was sent to us, we're appending, inserting, iterating, etc. over element trees. The data structure is very simple, how it looks when you serialize it is up to you - like the article, I also think tag soup looks horrible, so I serialize into a Python-style indented-block format when printing. I might get round to doing an input format as well.
Of course, it would be nice if the default serialization format sucked less.
Extending your point about tools, most people don't really have good XML editors. There's no reason I shouldn't be able to fire up 'vix' and delete, insert, etc. elements and attributes. In the Unix world in particular the tools are still based around presentation rather than structure.
Agreed. This is common to most text formats though. It This is why most XML apps use URL relationships to refer to binaries.
> Incredibly verbose
There's a great deal on confusion between XML the data structure and XML the common method of serializing that data structure. Probably because they have the same name.
Primarily, we deal with XML the data structure. If we create something from scratch, or once we de-serialize what was sent to us, we're appending, inserting, iterating, etc. over element trees. The data structure is very simple, how it looks when you serialize it is up to you - like the article, I also think tag soup looks horrible, so I serialize into a Python-style indented-block format when printing. I might get round to doing an input format as well.
Of course, it would be nice if the default serialization format sucked less.
Extending your point about tools, most people don't really have good XML editors. There's no reason I shouldn't be able to fire up 'vix' and delete, insert, etc. elements and attributes. In the Unix world in particular the tools are still based around presentation rather than structure.
I think the author of that article sucks. He got himself in trouble with me by saying his first complaint was XML doesn't handle binary data. wtf? Must be the amateur hour tonight.
As a standard, XML is actually pretty simple and, in my opinion, well thought out. Alas, freely available high quality documentation to take someone from 0 knowledge to a good grasp appears to be lacking. Many tools which claim to process XML are poorly written.
The author's list of complaints and some responses:
* "XML is very bad at handling binary data." That is mostly false. There is no obviously winner (widely adopted) standard for binary format XML although there will be some day. At that time, XML will be the best choice for new binary file formats and, I predict, processors for many legacy binary formats will gain DOM-based APIs. The surface syntax of textual XML requires that binary data be encoded into a safe character set. That's actually important for data interchange.
* "XML is incredibly verbose." Yes, yes it is. The primary sources of apparently needless verbosity are first, the need to mention the element name in closing end tags ("</foo>") and, second, the need to always wrap element names and closing tags (and attributes) in "<...>" pairs. The parent language (SGML) had mechanisms to relax those constraints and yield a more compact syntax. These were dropped in XML (for now) to make it easier to write parsers by getting the general case working first. Perhaps over time we will reintroduce a more flexible syntax or perhaps we'll discover we don't need it much.
* "DTDs use different syntax." That is a complaint about DTDs, not XML. Some schema languages use XML syntax and, as I recall, there is an XML syntax for DTDs. This complaint is mis-directed.
* "Confuses meta-data with content. Sometimes XML attributes contain values other times the value is in XML text." This statement is inaccurate and betrays the authors poor understanding of XML (see above about the poor state of XML documentation). Sometimes data is placed in element attributes and other times data is placed in XML sub-elements (not necessarily "text").
An XML datum is an inductively defined finite tree structure. Leaf nodes may contain text or more shallow trees. Leaf nodes are labeled with a type tag - the element name. The type of a node is parametric - it has parameters. Those parameters are element attributes. Generally speaking, if you have data that tells you something about the contents of an element, or about how to interpret the element - it goes in attributes. Otherwise, it goes into sub-elements.
A good criticism of XML would be that currently, attribute values may not be arbitrary XML datums - but there is nothing in the definition of XML that prevents fixing this in the future.
* "Unclear use of whitespace. Should a carriage-return after a tag be ignored? Should tabs be ignored?" This statement betrays the author's poor understanding of XML (see above about the poor state of XML documentation). In fact, XML specifications state two things that lead to the author's confusion, and some XML-based tools do confusing third things. (1) XML specifies a precise and simple canonicalization of whitespace. (2) XML specifies how to handle whitespace in tools which are manipulating XML source code rather than manipulating the canonical form. (3) Some tools do random, ad-hoc things - these are bad tools.
* "XML tools/libraries are very bad. The standard Java libraries have come a long way, but DOM parsing is still a chore." Many tools and libraries are very bad, some our wonderfully good. Because the author has a poor understanding of XML (see above about the poor state of XML documentation) he is not able to quickly tell the good tools from the bad and so the whole field looks to him a mess. I sympathize with the author quite a bit but if he understood XML better and stuck to good tools, his only complaint would be that there aren't enough good tools yet (though there are quite a few and you can get quite far with just what's already there).
The author complains: "XML has so many problems that an endless series of patches have been pasted on it. Each patch, however, has made XML worse: CDATA, namespaces, etc."
I am fairly certain that it is historically inaccurate to describe namespaces or CDATA as "patches pasted on".
Namespaces are, in fact, one of the great strengths of XML. The real problem is that too many programmers making use of XML don't understand them and therefore fail to use them or fail to use them properly (see above about the poor state of XML documentation). XML namespaces create a cooperative way to allocate element and attribute names in a distributed and decentralized way without introducing any central authorities not already present in the allocation of URLs. In this way, XML element names are vastly superior to lisp atom (symbol) names for tagging tree nodes and attribute names. Namespaces are one of the more important advances that XML represents.
I have less of a defense for CDATA but not no defense at all. On one level it is a handy addition to XML surface syntax. At the DOM level, it helps preserve the ability to reasonably "pretty-print" an XML datum. It is a bit ad hoc but it causes very little harm and is often enough quite convenient.
The author opines: "It was clear to me when XML first came on the scene that it was a bad specification for data interchange. How it came to be so widely used is a mystery to me. The most plausible explanation is that XML became a means for Microsoft, Sun, Oracle, et al to attempt to wield control over the industry. Each company used XML as a basis for creating semi-proprietary protocols that they could control."
XML has been adopted by far more players than Microsoft, Sun, and Oracle and is used by some quite excellent free software products. So many very talented programmers and teams have chosen XML that the author should consider applying Occam's razor and the scientific method here. A simpler explanation than his conspiracy theory is that the author's judgement that XML is a bad format is a mistaken judgement. (See above about the poor state of XML documentation.)
Finally, the author concludes: "The industry appears to be moving away from XML [....]" From where I sit, he is profoundly mistaken. Javascript toolkit authors and similar have often moved to JSON and similar, for temporary convenience, but in many other domains XML continues to make slow and steady advances, receiving huge amounts of investments.
I really, really do not want anything I've said above to be taken as a criticism of the author's person or skills. I don't mean to say that he is professionally wrong to shy away from XML in his own work. (See above about the poor state of XML documentation (and many tools).) His report is interesting as a reflection of frustrations that people in some domains are feeling. It's just mostly wrong on mostly every point (but for understandable reasons).