gcr makes a number of excellent points here. I completely agree about the human performance comments. Human performance on LFW is 99.2%. While some people claim this is "not fair" because it includes some background of the picture, this is the way the benchmark has been defined. There is no universally accepted way to define the face verification problem, and when we built LFW we chose to define the task on these images (without cropping them closely). Not surprisingly human performance goes down when the images are cropped more closely to the face. My response to people who say that it is unfair to use the whole image is "go ahead and write a program which takes advantage of the whole image!"
I have not read the paper carefully yet, so I cannot comment on the technical content. However, people have raised the question of why it's not yet on the LFW results page. First of all, we don't put up results until we received the results file from the authors yet, and we haven't received such a file yet, so of course the results are not up yet. Second, even if we receive results files from an author, we require that the paper either be published in a peer-reviewed conference or journal, or, alternatively, that if it is commercial, that an executable be available so that the system can be tried by other experimenters. If we are sent results from a paper that has been accepted but not yet published we sometimes show the results on our results page, but highlight it in red to show that the paper is forthcoming.
Additionally, Gary Huang, who does a fantastic job maintaining the LFW site, is redoing the entire results page to better present the information, and so results may experience a short delay while we are working on the new page.
One interesting thing to note about methods that are getting very high accuracy rates on LFW is that they typically use enormous amounts of outside training data (like the recent DeepFace method by FaceBook researchers). Part of the purpose of our reorganization of the results page is to highlight better the differences between methods that use outside data and those that use only the LFW training data.
The community should remember that a K-nearest neighbor classifier is the best possible classifier for any supervised learning problem when you have enough training data (as proven by Stone, 1977). Thus, it shouldn't be too surprising that many different methods work very well when given massive training sets. We saw the same thing back in the 90's with the MNIST data set.
I agree with gcr that moving to new data sets makes sense, although performance on LFW with limited training data may continue to be interesting for some time.
gcr makes a number of excellent points here. I completely agree about the human performance comments. Human performance on LFW is 99.2%. While some people claim this is "not fair" because it includes some background of the picture, this is the way the benchmark has been defined. There is no universally accepted way to define the face verification problem, and when we built LFW we chose to define the task on these images (without cropping them closely). Not surprisingly human performance goes down when the images are cropped more closely to the face. My response to people who say that it is unfair to use the whole image is "go ahead and write a program which takes advantage of the whole image!"
I have not read the paper carefully yet, so I cannot comment on the technical content. However, people have raised the question of why it's not yet on the LFW results page. First of all, we don't put up results until we received the results file from the authors yet, and we haven't received such a file yet, so of course the results are not up yet. Second, even if we receive results files from an author, we require that the paper either be published in a peer-reviewed conference or journal, or, alternatively, that if it is commercial, that an executable be available so that the system can be tried by other experimenters. If we are sent results from a paper that has been accepted but not yet published we sometimes show the results on our results page, but highlight it in red to show that the paper is forthcoming.
Additionally, Gary Huang, who does a fantastic job maintaining the LFW site, is redoing the entire results page to better present the information, and so results may experience a short delay while we are working on the new page.
One interesting thing to note about methods that are getting very high accuracy rates on LFW is that they typically use enormous amounts of outside training data (like the recent DeepFace method by FaceBook researchers). Part of the purpose of our reorganization of the results page is to highlight better the differences between methods that use outside data and those that use only the LFW training data.
The community should remember that a K-nearest neighbor classifier is the best possible classifier for any supervised learning problem when you have enough training data (as proven by Stone, 1977). Thus, it shouldn't be too surprising that many different methods work very well when given massive training sets. We saw the same thing back in the 90's with the MNIST data set.
I agree with gcr that moving to new data sets makes sense, although performance on LFW with limited training data may continue to be interesting for some time.
-Erik Learned-Miller