Google buys Eyefluence eye-tracking startup(techcrunch.com)
techcrunch.com
Google buys Eyefluence eye-tracking startup
https://techcrunch.com/2016/10/24/google-buys-eyefluence-eye-tracking-startup/
12 comments
The point of eye tracking in VR isn't really about being an input control (even if that's what this company was focusing on).
There are many big problems in VR that get solved if you know where the user is looking at. The most important one is, your eyes really only see high resolution in a very small region around what you're staring at, and if the VR headset can only really render that region in high quality, this solves many issues (processing power, framerate, even wireless transmission).
Of course, for any of this to work you need perfect eye tracking, which is very hard.
There are many big problems in VR that get solved if you know where the user is looking at. The most important one is, your eyes really only see high resolution in a very small region around what you're staring at, and if the VR headset can only really render that region in high quality, this solves many issues (processing power, framerate, even wireless transmission).
Of course, for any of this to work you need perfect eye tracking, which is very hard.
Either that, or most of the companies know that the technologies they are building are likely to be acquired to build part of the feature set of a larger offering, and are building them for precisely that reason. Too cynical?
Why is that cynical? That seems like a very reasonable thing to do if you want to start a company with very narrow domain knowledge.
I know a person that built a company for acquisition like you mention. Don't think they were ever successful. To be an acquisition target is one thing. But to be acquired by companies like Google most people would have to meet their hiring bar (aka passing their interviews) and have the right composition for the team (mostly engineers they would have a hard time hiring away). Of course there are exceptions, like they're solely purchasing for the technology, but that means the tech they built was pretty good and couldn't easily be cloned (of course this excludes if they're buying for market share of users if the product is super popular). My opinion is incredibly risky building for acquisition. Anyone have examples that it isn't?
This operates on the assumption that eye tracking and brain waves signals are positively or negatively correlated (for which the setup you propose can be thought of as an estimator with control variates). Is this indeed the case?
Eye movements and EEG are actually somewhat tricky to record and/or interpret simultaneously.
Eye movements cause travelling waves to sweep across visual brain areas (which are actually a huge chunk of the brain), potentially swamping whatever was going on before. Some of this is probably supports visual processing (by, e.g., persisting or resetting neural activity across eye movements), and some of it might be epi-phenomenal.
Furthermore, EEG electrodes pick up huge, non-neural signals from the movement of the ocular muscles and the movement of the eyeball itself. This can be reduced with clever pre-processing and more expensive amplifiers (e.g., that don't saturate or recover from saturation more quickly).
Eye movements cause travelling waves to sweep across visual brain areas (which are actually a huge chunk of the brain), potentially swamping whatever was going on before. Some of this is probably supports visual processing (by, e.g., persisting or resetting neural activity across eye movements), and some of it might be epi-phenomenal.
Furthermore, EEG electrodes pick up huge, non-neural signals from the movement of the ocular muscles and the movement of the eyeball itself. This can be reduced with clever pre-processing and more expensive amplifiers (e.g., that don't saturate or recover from saturation more quickly).
I'm not sure what information you're working off that makes you think they need to be positively correlated. I'm referring to using two separate systems in conjunction. Eye tracking to cursor placement (and anything else that it can handle well), and brainwave tracking to handle intent. At the simplest level, eye tracking for mouse x/y coordinates, and brainwave tracking for mouse button presses.
For those wondering about the privacy implications of this technology, eyefluence required a VR headset to function and I'm not aware of them demonstrating their tech using the user-facing cameras found in mobile phones and tablets, so I don't think there is any reason for worry on that front (yet).
Here is a video of a 2016 demo of eyefluence tech:
https://www.youtube.com/watch?v=hYH8qLvq7rc
Here is a video of a 2016 demo of eyefluence tech:
https://www.youtube.com/watch?v=hYH8qLvq7rc
Would this be interesting for Google's Daydream efforts?
10 years ago while I was IBM, I submitted a proposed patent that was a sort-of-joke by saying Amazon (or other sites) could get rid of 1-click shopping and go to 0-click shopping by following your eyes and automatically making the purchase if you looked at something for a given amount of time. Sounds like this will be a reality in a few years.
Wife: Honey, do you know why Amazon keeps sending us mini-skirts and fishnet stockings? I mean, they're not even in my size!
Husband: Nope, sorry, no idea!
Husband: Nope, sorry, no idea!
Was it accepted?
(Serious version of this: Amazon could feed eye-tracking data into their "stock a product near the user if they're likely to buy it" algorithm.)
(Serious version of this: Amazon could feed eye-tracking data into their "stock a product near the user if they're likely to buy it" algorithm.)
How many people here cover the front-facing camera on their phone when they're not using it? I haven't, until now.
The Samsung phones have a ton of eye and face detection options for brightness and screen wake up based on eye and face detection, which I always turn off to conserve battery. I have never heard of that feature on iPhone though.
Physically removed both front and back integrated cameras from my tablet. If the FBI Director is paranoid enough to cover his webcam then that is good enough reason for drastic measures...
Privacy aside, I have an ugly face.
Privacy aside, I have an ugly face.
This does sound like a privacy disaster though they don't mention smartphones in the article ?
What does sound like a problem is eyes for something they're not intended for, being hands and fingers, I wonder if people will end up with RS'eye' ;)
What does sound like a problem is eyes for something they're not intended for, being hands and fingers, I wonder if people will end up with RS'eye' ;)
Do you also now cover your computer mic?
This is pretty much why I keep the phantom power off when my condenser mic isn't in use. Literally can't work unless I flip a physical switch.
Not trolling, serious question: What is the audio/mic equivalent of covering your webcam with a sticker?
Use a machine without an embedded mic (neither of my desktop machines have one)
One that I've seen is plugging something into the audio jack, and then breaking it off and taping over it. That way the computer will detect something is plugged in and turn off the mic. Of course this won't be possible once Apple, and then everyone else, gets rid of the audio jack from computers.
Isn't that still software controlled? So, let's say a malware is able to gain privileged access to my Windows machine and can install anything, would he be able to override this default behavior of 'turning off the mic when something is plugged in', especially when the external voices are more audible than something coming from the 'plugged in' dummy audio-in cable?
It would have to be an audio device that has that little "ring" of audio in. Sorry, not knowledgeable in this field, someone smarter will be able to clarify, but I've noticed that my headphones that don't have a mic capability have less little "separator" rings on the 3.5mm jack, whereas the ones that do have mic capability have more.
Also your solution seems... excessive...
Also your solution seems... excessive...
https://www.obdev.at/products/microsnitch/index.html (best you are going to get at a software level).
Ideally, get a system with a microphone killswitch, such as the Purism devices.
Whether you do that or not, use an OS that you trust to mute the microphone, and don't run untrusted applications outside a sandbox.
Whether you do that or not, use an OS that you trust to mute the microphone, and don't run untrusted applications outside a sandbox.
unplugging it
I presume GP means the built-in mic that sits next to your built-in webcam.
Better details from a beta tester over on /r/oculus on Eyefluence's tech:
" . . . I'm still not allowed to say how exactly it works but in my mind it's the gold standard for eye based user interfaces . . . Their [main focus] is the UI" (1)
So sounds like a blend between some good eye tracking tech and some GREAT user experience design and implementation.
(1) https://www.reddit.com/r/oculus/comments/598clo/google_buys_...
" . . . I'm still not allowed to say how exactly it works but in my mind it's the gold standard for eye based user interfaces . . . Their [main focus] is the UI" (1)
So sounds like a blend between some good eye tracking tech and some GREAT user experience design and implementation.
(1) https://www.reddit.com/r/oculus/comments/598clo/google_buys_...
How difficult is it to track eyes using modern deep learning techniques? Isn't this a simple, run of the mill task by now?
I'm a machine learning researcher and I also have a lot of experience analyzing eye movement data.
There aren't large labeled datasets that are publicly available in this area. Just to classify eye movements into events, you would need images of the eyes or eye velocity to classify eye events into smooth pursuit, fixation, etc. That means someone needs to label all of those in ground truth. To actually determine what someone is looking at, you have to go beyond this to have information about the scene. When looking at a screen, this is usually done with calibration, but things get more complicated when trying to predict what someone is looking at with a mobile eye tracker and a fixed scene camera. Body and head movements complicate things further (e.g., VOR) and many algorithms ignore them.
There aren't large labeled datasets that are publicly available in this area. Just to classify eye movements into events, you would need images of the eyes or eye velocity to classify eye events into smooth pursuit, fixation, etc. That means someone needs to label all of those in ground truth. To actually determine what someone is looking at, you have to go beyond this to have information about the scene. When looking at a screen, this is usually done with calibration, but things get more complicated when trying to predict what someone is looking at with a mobile eye tracker and a fixed scene camera. Body and head movements complicate things further (e.g., VOR) and many algorithms ignore them.
My research group has been trying to address this problem by using user interactions with a website to create this "ground truth". One assumption is that you have to look where you click. But there are more subtle user interactions that could be helpful.
Video and demos and source code: http://webgazer.cs.brown.edu/
Video and demos and source code: http://webgazer.cs.brown.edu/
It's a good start, but nowhere near as accurate in comparison to the infrared trackers, though? What would it take to get it to that level?
Depends on which aspect you mean. E.g. a proper infrared tracker with high framerate, something like >120Hz, can be used to detect microsaccdes. I don't know much about webcam tech, but I doubt it can reach those framerates and at the same time low noise needed to distinguish those microsaccades. Then again, the research of the meaning of those and whether they're predictive or not isn't conclusive afaik so I wouldn't know any use for VR anyway.
If you don't mind me jumping in here, since people with eye tracking experience seem to be quite scarce: is it viable these days to build a working accurate gaze tracker that would, say, work using just an iPad camera? No hardware whatsoever, just using the camera to somewhat accurately position a dot on the screen. Every tracker I've tried is close to absolute garbage when it comes to accuracy.
YC invested in a company called 'GazeHawk', but they didn't make it (acquihire by facebook), the page is still up though.
http://www.businessinsider.com/facebook-acquires-gazehawk-fo...
http://www.gazehawk.com/
http://www.businessinsider.com/facebook-acquires-gazehawk-fo...
http://www.gazehawk.com/
Yeah.. I've literally tried 20 solutions, and most can not detect gaze location well. Gazehawk included.
That may have been one of the reasons why they didn't make it.
Eye tracking is tricky. I worked a bit with two guys developing a commercial eye tracker in the 80's, it was a pretty sophisticated optical set-up that combined the image being looked at and the image of the person looking using a 45 degree angle 50% mirror, that made the job considerably easier.
One of the complicating factors was that eye movements have a lot of noise in them that our brain has become very adept at filtering out so what we think our eyes are doing can be quite different from what they're really doing.
Eye tracking is tricky. I worked a bit with two guys developing a commercial eye tracker in the 80's, it was a pretty sophisticated optical set-up that combined the image being looked at and the image of the person looking using a 45 degree angle 50% mirror, that made the job considerably easier.
One of the complicating factors was that eye movements have a lot of noise in them that our brain has become very adept at filtering out so what we think our eyes are doing can be quite different from what they're really doing.
Not easily. Samsung has something like this, which attempts the minimal task of "see if you're looking at the screen" to prevent dimming while you read. It's... not good. Not even passable, in my experience.
That task should be achievable, but anything fancier - even positioning a dot - is probably still a tall order with that single camera.
That task should be achievable, but anything fancier - even positioning a dot - is probably still a tall order with that single camera.
The hardware based IR solutions (e.g. Tobii) tend to work really well, though. Is it a matter of not having figured out yet how we could achieve similar results using just a webcam image?
The IR part is the key--webcams have IR-cut filters to keep the visible-spectrum picture from looking crappy.
I was wondering if you could use such a device with a small laser to get where the eyes points, it would also give velocity : https://www.youtube.com/watch?v=BZjtufbRZwA
Lasers and eyes don't mix very well. A single stray reflection off something spherical and you could end up with serious eye damage.
Warning sign on a friends watercooled laser: 'do not look into laser with remaining eye'.
Warning sign on a friends watercooled laser: 'do not look into laser with remaining eye'.
You don't need deep learning. Seriously. Not everything in the world needs deep learning. I wrote an eyetracking system for tracking my vision while playing mobile games[0], and really, the eye-tracking is quite simple and you can do it with violajones haar cascade and do it with an accuracy high enough that you can press buttons with your vision.
The hardest part were ellipse curve-fitting algorithms - I had to go pretty cutting edge for that.
[0] http://blog.chewxy.com/2014/04/08/eyetracking-jetpack-joyrid...
The hardest part were ellipse curve-fitting algorithms - I had to go pretty cutting edge for that.
[0] http://blog.chewxy.com/2014/04/08/eyetracking-jetpack-joyrid...
At the last Oculus Connect, Michael Abrash covers some of the high level issues they've seen with fast and "virtually perfect" eye tracking that is required for VR [1]. He predicts it should be doable within the next 5 years (including new rendering techniques required for foveated rendering).
His entire keynote is great, particularly if you're interested in the near-future possibilities of VR.
[1] https://youtu.be/AtyE5qOB4gw?t=13m52s
His entire keynote is great, particularly if you're interested in the near-future possibilities of VR.
[1] https://youtu.be/AtyE5qOB4gw?t=13m52s
Well, they're not just tracking eyes: they are using eye position, focus, and gestures to control a user interface which is a bit different.
The moment the stuff you want to track (eyes) also makes the enviroment change (ui) stuff can get tricky.
Eye blinking also changes the way we perceive stuff. If we use our eyes to control a UI, blink events are also special because they usually happen when we have processed enough information from a scene (Walter Murch book: In the blink of an eye tells something about this from a filmmaker perspective).
So... not as easy as it looks at first sight (no joke intended ;-).
The moment the stuff you want to track (eyes) also makes the enviroment change (ui) stuff can get tricky.
Eye blinking also changes the way we perceive stuff. If we use our eyes to control a UI, blink events are also special because they usually happen when we have processed enough information from a scene (Walter Murch book: In the blink of an eye tells something about this from a filmmaker perspective).
So... not as easy as it looks at first sight (no joke intended ;-).
When I wrote EyeMap, it turned out humans blinked a lot more than I had anticipated. It wasn't much later until EyeMap had been shuttered before I read that same book. What I did was basically ignore the 5 or so frames where the eye was blinking because EyeMap's main purpose wasn't to be a UI - the button pressing bits are only for calibration purposes.
Basically? It's still hard.
Identifying eyes is fairly easy, of course. Identifying open, approximately centered eyes is the easiest version of this and it's what we want.
Basic eye-tracking efforts say something like "which of these equidistant squares did your line of vision intercept?" That's still enough to produce some cool things like "look at the red box to click it", but it's not a real-world solution.
But the problem is identifying small changes and cross-referencing them with the state of the world to identify what's actually being viewed. The difference between someone looking at a sticker on a window and looking out that window is big, but the evidence available to distinguish them is tiny.
Other 'natural' movements can also spell doom for eye-tracking: turning your head without moving your eyes sweeps your vision, while turning your eyes saccades it and only values the endpoints. And turning your head and body while keeping focused on a single point feels simple, but identifying it means tracking three different axes of rotation as they interact!
Identifying eyes is fairly easy, of course. Identifying open, approximately centered eyes is the easiest version of this and it's what we want.
Basic eye-tracking efforts say something like "which of these equidistant squares did your line of vision intercept?" That's still enough to produce some cool things like "look at the red box to click it", but it's not a real-world solution.
But the problem is identifying small changes and cross-referencing them with the state of the world to identify what's actually being viewed. The difference between someone looking at a sticker on a window and looking out that window is big, but the evidence available to distinguish them is tiny.
Other 'natural' movements can also spell doom for eye-tracking: turning your head without moving your eyes sweeps your vision, while turning your eyes saccades it and only values the endpoints. And turning your head and body while keeping focused on a single point feels simple, but identifying it means tracking three different axes of rotation as they interact!
Even though this is for hardware and VR, I have to admit the first thought on seeing the headline was that google doubled down on its advertisement business. More so after the recent news that they changed their policy to track users more.
He asks us, "how do you transfer your intent to act without winking and without waiting", but never tells us how its done. Anyone know? This made the video incredibly frustrating for me.
They use saccades + some fancy algorithms: https://en.wikipedia.org/wiki/Saccade
That was super frustrating, I went back a few times to see if he said it and he did not. I was just waiting for him to tell me for the rest of the video and it never happened.
Facebook bought an eye-tracking startup a few years back: http://www.businessinsider.com/facebook-acquires-gazehawk-fo...
At the time, i read it was only a acquihire
At the time, i read it was only a acquihire
By the way, Eyefluence's marketing videos are brilliant. I've seen people's arm hairs stand straight watching them.
They'd be worth acquiring if only for the marketing department.
They'd be worth acquiring if only for the marketing department.
I can see good eye tracking as getting much, much more traction than VR for regular users. While most applications of VR to everyday tasks seem forced and gimmicky, eye tracking can revolutionize reading, which is arguably the task professional and casual users spend at least 80 percent of their online time on.
Much smarter than investing in VR.
Much smarter than investing in VR.
The two main cases I'm thinking of are eye tracking, and brainwave tracking (like EMOTIV). Individually, neither looks like a compelling way to control a computer when you have a mouse and keyboard, but together, I think that might really yield something interesting (and sooner!).
Instead of using brain waves to move a cursor around the screen, use eye tracking. Instead of using gaze lingering for click, use brainwave tracking. Individually they seem cumbersome and annoying to use. Together they could be a really compelling interface device, IMO.