Learning Word Vectors for 157 Languages(arxiv.org)
arxiv.org
Learning Word Vectors for 157 Languages
https://arxiv.org/abs/1802.06893
8 comments
I argue that you need both domain-general and domain-specific data. Most domains you could apply NLP to don't have billions of words to train on, especially if the language of the data isn't English.
Pre-trained data is valuable, and you really aren't going to re-learn it all from your data, so why throw it out?
Pre-trained data is valuable, and you really aren't going to re-learn it all from your data, so why throw it out?
Yes, recent work on transfer learning https://arxiv.org/abs/1801.06146 " Fine-tuned Language Models for Text Classification" suggests that training a lstm on a large corpora for language modeling, then using it as sentence encoder after tuning it in specific dataset gives a considerable perfomance.
This is covered in fast.ai 's new course 1.
This is covered in fast.ai 's new course 1.
I'm always amazed that, although Cantonese is about as commonly spoken as Italian, so little written corpus data exists for it (since most Cantonese write in a form of Mandarin Chinese).
For example, here there are all kinds of useful things we can do with these 157 sets of word vectors, but Cantonese escaped the list because most of its transactions happen off the page.
https://en.wikipedia.org/wiki/Written_Cantonese
For example, here there are all kinds of useful things we can do with these 157 sets of word vectors, but Cantonese escaped the list because most of its transactions happen off the page.
https://en.wikipedia.org/wiki/Written_Cantonese
There seems to be some interest in learning what word vectors are, so I'll give a basic overview here, then tie it into the research.
"One of the challenges for Natural Language Processing (NLP) systems is the question of how to represent input such that the network runs quickly, but also learns well. It's possible to represent each word as a one hot vector, but that's computationally slow. It's also possible to represent each word as some number, but then lots of words look very similar.
Instead, why not use a mix? Introducing word embeddings. We'll represent each word as a n-dimensional vector, with each dimension representing a trait about the word. For example, "fruit" might be represented as {food: 0.99, gender: -0.05, size: 0.2}, and "king" might be represented as {food: -0.9, gender: 0.92, size: 0.56}." [Quoted from MuffinTech.org] [See v1n337 for caveats. [0]]
Two similar words should have similar word vectors, like "apple" and "peach". If we learn some fact about apples, like "Humans eat apples.", then we can easily generalize that to peaches, pineapples, etc...
Let's tie this back to the research. Since we have word vectors for many languages now, that makes it easier for us to build NLP systems in other languages. For example, if we wanted to build an English->French translator.
[0] https://news.ycombinator.com/item?id=16448960
"One of the challenges for Natural Language Processing (NLP) systems is the question of how to represent input such that the network runs quickly, but also learns well. It's possible to represent each word as a one hot vector, but that's computationally slow. It's also possible to represent each word as some number, but then lots of words look very similar.
Instead, why not use a mix? Introducing word embeddings. We'll represent each word as a n-dimensional vector, with each dimension representing a trait about the word. For example, "fruit" might be represented as {food: 0.99, gender: -0.05, size: 0.2}, and "king" might be represented as {food: -0.9, gender: 0.92, size: 0.56}." [Quoted from MuffinTech.org] [See v1n337 for caveats. [0]]
Two similar words should have similar word vectors, like "apple" and "peach". If we learn some fact about apples, like "Humans eat apples.", then we can easily generalize that to peaches, pineapples, etc...
Let's tie this back to the research. Since we have word vectors for many languages now, that makes it easier for us to build NLP systems in other languages. For example, if we wanted to build an English->French translator.
[0] https://news.ycombinator.com/item?id=16448960
for the uninitiated, an 'n-dimensional vector' is an array of numbers
If Apple and peach had very similar word vectors, an English apple and a French peach would have the same too. And there is a risk of mistranslatuon. How is that situation handled?
It is handled by supervised training with paired translations, so that English apple will be associated with French pomme instead of other fruits. If you don't have a parallel corpus, translation gets significantly harder. I'm actually more amazed that it's possible at all.
And what about meanings in various contexts like financial content with words Apple, Orange?
If you know beforehand what kind of document you are dealing with, you can refine the word vectors for your given task, or even train them from scratch if you have enough data. In general, though, you'll end up with a mixture of meanings. The vector for "Apple" would be somewhere between fruits and companies, while the vector for "Amazon" would be somewhere between rivers and companies.
An interesting paper looked at how these associations changed over time [1]. It was also featured recently on The Morning Paper [2], in case you prefer a summary with added context.
Although those ambiguities make things a bit more difficult, you can usually leave the job of disentangling them to a later stage in the language-modeling process, which will have more context it can use to disambiguate which word sense was used.
[1] https://arxiv.org/abs/1703.00607
[2] https://blog.acolyer.org/2018/02/22/dynamic-word-embeddings-...
An interesting paper looked at how these associations changed over time [1]. It was also featured recently on The Morning Paper [2], in case you prefer a summary with added context.
Although those ambiguities make things a bit more difficult, you can usually leave the job of disentangling them to a later stage in the language-modeling process, which will have more context it can use to disambiguate which word sense was used.
[1] https://arxiv.org/abs/1703.00607
[2] https://blog.acolyer.org/2018/02/22/dynamic-word-embeddings-...
That is a slight problem. Disambiguates start to dive into higher contextual meaning where we need to look at nearby words. This means there are likely some word vectors whose meanings are "muddled", per se.
Although, I suppose if we treat "apple" and "Apple" as different words, that would help.
Fun fact: One of the current NLP problems is detecting which words are names. Apparently it's really tough, especially with Twitter data!
Although, I suppose if we treat "apple" and "Apple" as different words, that would help.
Fun fact: One of the current NLP problems is detecting which words are names. Apparently it's really tough, especially with Twitter data!
I suppose that if you're doing multilanguage, this problem partially sorts itself out. E.g. Spanish there will be Apple and manzana, in two different places due to their different semantics. Now for English, say you were trying to place "apple" in that space, you would want to put it next to both of them.
Unfortunately I see a problem in having to specify an exact position per word. If you think of the position of english "Apple" in the Spanish word space as a distribution instead of a specific location, then it ideally should be a two-mode distribution, with one peak next to Apple and one peak next to manzana. If you must use a normal distribution, the variance must be wide enough to cover both words -- a huge problem, since (a) that assigns a lot of probable values to one word and (b) the mean value (expected value) lies between them, not at the semantic location of "apple" at all.
Unfortunately I see a problem in having to specify an exact position per word. If you think of the position of english "Apple" in the Spanish word space as a distribution instead of a specific location, then it ideally should be a two-mode distribution, with one peak next to Apple and one peak next to manzana. If you must use a normal distribution, the variance must be wide enough to cover both words -- a huge problem, since (a) that assigns a lot of probable values to one word and (b) the mean value (expected value) lies between them, not at the semantic location of "apple" at all.
If you are interested in multi-language text analysis, you might be interested in Freeling [1],
a full-fledged, open source library for language analysis written in C++ (which also happens to include a simple interface for working with word vectors).
[1] http://nlp.lsi.upc.edu/freeling/node/1
[1] http://nlp.lsi.upc.edu/freeling/node/1
Online-reader-friendly (no 2-column PDF!) paper link:
https://www.arxiv-vanity.com/papers/1802.06893/
https://www.arxiv-vanity.com/papers/1802.06893/
Newbie here. Can someone explain what would I use these vectors for?
Besides what others have already said: word vectors are usually calculated from the context they appear in, the other words surrounding them. The vector representation can be intuitively understood as representation of the context the word appears in. The number of words we take into account for that context is called 'window'.
This has very interesting possibilities: you can complete sentences where a word is missing (you have a context, so you can search for the best matching word vector), use it in text autocorrection tasks, and other classic natural language processing problems.
Learned word representations are also coherent between them, so you can use them to make analogies (the distance from 'Spain' to 'Madrid' is similar to the distance from 'France' to 'Paris'), so they implicitly hold some of the semantic info between words.
It can also be used to find related words. Synonyms, antonyms and related words have similar representations. For example, 'facebook', 'twitter' and 'instagram' have similar vectors (vectors with similar directions). But you can also try with famous musician or band names, tech related terms, etc.
Finally, word vectors, unlike other language models, can store representations for large windows and vocabulary sizes in a few GB, which is another useful property in certain situations, and makes them easy to handle.
This has very interesting possibilities: you can complete sentences where a word is missing (you have a context, so you can search for the best matching word vector), use it in text autocorrection tasks, and other classic natural language processing problems.
Learned word representations are also coherent between them, so you can use them to make analogies (the distance from 'Spain' to 'Madrid' is similar to the distance from 'France' to 'Paris'), so they implicitly hold some of the semantic info between words.
It can also be used to find related words. Synonyms, antonyms and related words have similar representations. For example, 'facebook', 'twitter' and 'instagram' have similar vectors (vectors with similar directions). But you can also try with famous musician or band names, tech related terms, etc.
Finally, word vectors, unlike other language models, can store representations for large windows and vocabulary sizes in a few GB, which is another useful property in certain situations, and makes them easy to handle.
At one level, it’s simply a vector of weights. In a simple 1-of-N (or ‘one-hot’) encoding every element in the vector is associated with a word in the vocabulary. The encoding of a given word is simply the vector in which the corresponding element is set to one, and all other elements are zero. It's all about Natural Language Processing.
If you are interested in more, check out these excellent reviews by Adrian Colyer posted in The Morning Paper.
https://blog.acolyer.org/2016/04/21/the-amazing-power-of-wor...
If you are interested in more, check out these excellent reviews by Adrian Colyer posted in The Morning Paper.
https://blog.acolyer.org/2016/04/21/the-amazing-power-of-wor...
that's a nice link but the excerpt you quote is kind of misleading; the word vectors in this case are not one-hot encodings. They are learned, continuous representations. But one-hot representations are also a kind of word vector.
word vectors are vector representations of each word in the vocabulary. Here they are learned by a neural net. the length of the vector is the # of features. Just for intuition, one feature of a word the NN could learn is the gender of a word, and so on.
word vectors are vector representations of each word in the vocabulary. Here they are learned by a neural net. the length of the vector is the # of features. Just for intuition, one feature of a word the NN could learn is the gender of a word, and so on.
To help with intuition, here are a few example word vectors that we might encounter:
"fruit": {food: 0.99, gender: -0.05, size: 0.2}
"king": {food: -0.9, gender: 0.92, size: 0.56}
Building off of what v1n337 stated, though, axis can easily be skewed and rotated such that they're still interpretable, just not obviously so.
"fruit": {food: 0.99, gender: -0.05, size: 0.2}
"king": {food: -0.9, gender: 0.92, size: 0.56}
Building off of what v1n337 stated, though, axis can easily be skewed and rotated such that they're still interpretable, just not obviously so.
Yes, quite correct! The vectors are dense representations.
But the features aren't individually interpretable, in practice. For instance, the 'gender' of a word may have it's signal scattered over several features/dimensions of the learned vector.
But the features aren't individually interpretable, in practice. For instance, the 'gender' of a word may have it's signal scattered over several features/dimensions of the learned vector.
And corresponding word vectors: https://fasttext.cc/docs/en/crawl-vectors.html
Are these the right vectors? The filenames correspond to the fastText vectors I've already tried, which are only in English.
EDIT: Indeed, this is old data from a previous publication. It appears they have not actually made the new data public yet.
EDIT: Indeed, this is old data from a previous publication. It appears they have not actually made the new data public yet.
Update: that link points to the right page now.
I guess I can't argue with the results, but the data doesn't look very clean. I downloaded the Esperanto file because I expected it to be small, but it was a 1.1 GB download that expanded to 4.4 GB. A lot of the most popular "words" are punctuation marks, and the first two real words are "la" and "La". Perhaps naively, I expected those to be the same word.
We decided to keep the casing, as it is useful for some applications such as named entity recognition.
Regarding the punctuation, as pointed out in another comment, these tokens might also be useful for some applications (and they are easy to filter out if you don't need them).
Regarding the punctuation, as pointed out in another comment, these tokens might also be useful for some applications (and they are easy to filter out if you don't need them).
In the Tagalog file, } is near the top but { is over 8,000 lines down. Is there a reason they have such different frequencies? ( and ) are right next to each other.
And yes I realize this is a really odd question :)
And yes I realize this is a really odd question :)
This is probably due to our preprocessing of Wikipedia that did not get rid of all the '}' from the markup.
Oh true. I tried to clean up Wiki markup for ML years ago and it was a huge pain. Next time I think I'll parse the HTML version and pull out the text from the tags explicitly.
This is a much better way to do it. It's easier, cleaner, and gets the text which is generated by templates, which there is a surprising amount of (you get weird artifacts from that otherwise).
Your comment has twice as many ) as it does (
My first guess would be emojis ;)
My first guess would be emojis ;)
Does a word-vector dataset that "looks clean" perform better on real applications, justifying the extra attention required for more preprocessing/cleanup?
[deleted]
Word vectors are fascinating representations. There is a huge amount of information and nuance captured in them. You can use them directly for topic retrieval (using annoy or another optimised vector index), or feed them into a classifier such as those in the Sklearn library. All types of neural nets: fully connected, recurrent and convolutional can be applied on word vectors.