"Since faces are unique, “significantly altering a face to make it unrecognizable is difficult,” and names are distinctive, why do many people seem unconcerned about their public dissemination? The answer is simple. The norms governing our attitudes toward the name-face connection developed during time periods when it was hard to identify most strangers."
Where norms and technology abut - there you'll find privacy challenges.
In the U.S., implementation of a wiretapping scheme like this would be a significant civil and criminal violation. Google is already under a FTC consent degree for privacy violations, which would make it doubly egregious. So I'll give Google the benefit of the doubt - and assume there are privacy-protective mechanisms designed in to the system.
In the age of ambient technology, where anything can be recorded, the onus is now on developers to create systems that internally suppress mass privacy violation. The "Privacy by Design" approach (disclosure, I have an evangelist of PbD) can provide solid guidance as to how to build privacy-protective mechanisms (e.g. data minimization, data scrubbing) into ambient technologies.
Where norms and technology abut - there you'll find privacy challenges.