Parsing huge XML files with Go(blog.davidsingleton.org)

44 pointsby dpsvor 14 Jahren17 comments

blog.davidsingleton.org

http://blog.davidsingleton.org/parsing-huge-xml-files-with-go/

5 comments

Personally, I have a penchant for writing my own pull parsers. Its a mind-expanding exercise.

The neat thing about Go is that parsers can return functions that consume the next token. Rob Pike has an excellent video about this: http://www.youtube.com/watch?v=HxaD_trXwRE

signa11vor 14 Jahren

> Rob Pike has an excellent video about this: http://www.youtube.com/watch?v=HxaD_trXwRE

thank you ! this is an excellent talk. having concurrent implementation of lexer & parser as co-routines communicating over message channels is very, very cool.

pjmlpvor 14 Jahren

As in any language that supports functions as first class objects.

fleitzvor 14 Jahren

Streaming parsers are key when dealing with XML files this big. Used to have a C# parser that would parse about 1 TB of XML per day the biggest files were > 200GB.

It was impossible with out rewriting everything to use a SAX style parser.

barrkelvor 14 Jahren

SAX style (parsing library callbacks) is not your only option; you can also use an iterator style (i.e. something like XmlReader in C#).

fleitzvor 14 Jahren

Oops, mistook SAX for iterator, I really prefer XmlReader to SAX style.

masklinnvor 14 Jahren

> you can also use an iterator style (i.e. something like XmlReader in C#).

This category is generally called "pull parsers" (as opposed to SAX-style event-driven parsers)

Jimmievor 14 Jahren

I'm curious, what data did the XML files contain?

duanebvor 14 Jahren

As much as I like hearing about Go, SAX parsers are not exactly new.

iandvor 14 Jahren

I think showing the convenience of parsing into tagged structs makes this a cut above the usual SAX parsing examples.

duanebvor 14 Jahren

Hm, good point. Negativity redacted.

eximvor 14 Jahren

In the first place, why should you have huge XML files? (Except those wikipedia dump files :))

human_errorvor 14 Jahren

It happens sometimes. I had written a multithreaded parser in C++ to parse around 800MB per day so another team could build up the rest of the project based on the data. Someone had thought it'd be a better idea to store all fetched data in XML.

willvarfarvor 14 Jahren

XML is often used when migrating datasets, large and small. Interchange between disparate systems is the very thing its good for.

archangel_onevor 14 Jahren

OpenStreetMap is another example that uses huge XML files. I'm not sure I really like the idea, but it does happen, and if you need the data then you have to be able to deal with it somehow even if you don't like the format.

mercurialvor 14 Jahren

I had to deal with relatively large XML dumps containing dictionary data.

pradeepprabakarvor 14 Jahren

I had to do a similar task of parsing the huge wikipedia dump and rewriting the Wikipedia XML (I had to add a couple of other tags to the main "page" tag) I used a SAX parser in Python and rewrote the dump. I found SAX parsers very simple to deal with huge XML Streams.