Deep Learning, NLP, and Representations(colah.github.io)
colah.github.io
Deep Learning, NLP, and Representations
http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/
4 comments
Everything on this guy's blog is fantastic. I love the post on neural network manifolds (http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/).
When I experimented with word vector representations (word2vec), I found that the analogous embedding part was a bit oversold in a subtle way. While some sorts of analogies worked (the easiest to see was gender: woman-man+boy=girl), other sorts of analogies that appeared just as straightforward to a human were not at all understood by the embeddings.
For example,
water-boat+plane=potable_water (air, sky, etc nowhere to be found).
small-big+tall=taller (short, etc nowhere to be found).
shell-pecan+coconut=shells (husk nowhere to be found).
What I eventually realized is that the vector representations only really work in the case where an averaging operation would get the right answer anyways. For example, woman-man+boy= girl, but you can also get that through (woman+boy)/2=girl. (or even woman+boy=girl) In my experimentation I have yet to see a case where the subtraction aspect is actually necessary to recover the relationship. This would indicate that the vectors are only capturing similarity between words, and not and not any second order relationships between them as is commonly claimed.
Honestly, I wish I could discuss this directly with someone doing research in this area, because it seems to me that the vector representations really aren't capturing relationships in the way they're touted. I think it's an interesting possibility, and I haven't seen it addressed in any of the literature I've read.
EDIT: On further investigation, it looks like the google news vectors maybe just aren't that good? I wasn't able reproduce hardly any of the analogy examples given in the table "Relationship pairs in a word embedding." from the OP.
For example,
water-boat+plane=potable_water (air, sky, etc nowhere to be found).
small-big+tall=taller (short, etc nowhere to be found).
shell-pecan+coconut=shells (husk nowhere to be found).
What I eventually realized is that the vector representations only really work in the case where an averaging operation would get the right answer anyways. For example, woman-man+boy= girl, but you can also get that through (woman+boy)/2=girl. (or even woman+boy=girl) In my experimentation I have yet to see a case where the subtraction aspect is actually necessary to recover the relationship. This would indicate that the vectors are only capturing similarity between words, and not and not any second order relationships between them as is commonly claimed.
Honestly, I wish I could discuss this directly with someone doing research in this area, because it seems to me that the vector representations really aren't capturing relationships in the way they're touted. I think it's an interesting possibility, and I haven't seen it addressed in any of the literature I've read.
EDIT: On further investigation, it looks like the google news vectors maybe just aren't that good? I wasn't able reproduce hardly any of the analogy examples given in the table "Relationship pairs in a word embedding." from the OP.
The maths works when you are acting along the same vector(s), so things like king:male ~= queen:king work. In that example you are doing maths along the 'gender vector'.
In you water/boat/plane example I'm not sure what you are expecting. It seems unlikely that there is a 'planeiness' vector one can move along to find concepts like that.
In you water/boat/plane example I'm not sure what you are expecting. It seems unlikely that there is a 'planeiness' vector one can move along to find concepts like that.
I recommend you read the paper here:
http://nlp.stanford.edu/projects/glove/
It gives intuition/light math why the analogy as translation in embedding space should work.
http://nlp.stanford.edu/projects/glove/
It gives intuition/light math why the analogy as translation in embedding space should work.
I've read that multiplication/division works better than addition/subtraction. Like king/man\*woman, or something like that.
Always excellent content from Christopher Olah. So few people do such a clear job of explaining these concepts -- this is technical writing done right.
[deleted](1)