Tokenising the english text of 30TB common crawl(matpalm.com)
matpalm.com
Tokenising the english text of 30TB common crawl
http://matpalm.com/blog/2011/12/10/common_crawl_visible_text/
4 comments
Very good to see an example of working with the common crawl. I would encourage YC applicants to think about what kind of opportunities it presents.
Very interesting. Surprising to see Lithuania as the second most popular language. Oddly enough, I was having a conversation with someone this weekend when they mentioned that Lithuania had the highest fiber-to-the-home penetration in Europe and one of the fastest average internet connections in the world. Does anyone know why this is?
I once tried to identify the language in (supposedly) English-language Facebook posts. The second-most frequent language was Estonian. Looking at actual posts classified as Estonian, it turned out they contained text like "Soooooo nice". Because the language identifier looks at character sequences, and Estonian tends to have unusually common double vowels, such posts were classified as Estonian. Dutch and Norwegian were other two languages for which English was mistaken - apparently some character sequencies there have frequency distributions that are similar to English.
I think the Lithuania bit is a false alarm. The pipeline needs a lot more work and investigation of results. The main thing I was focusing on first was scaling it out.
You need how much right now, 200TB tops? including index files, uncompressed data and replication. With some clever compression and data structures you can probably cut that by half or less. Financially speaking this is already on local club territory.