Patent Filings Reveal the Untapped Potential of Google and Amazon Smart Devices

Google’s and Amazon’s virtual assistants could become scary smart.

The tech companies have explored how smart speakers like the Google Home, Amazon Echo, and other devices could monitor more of what we say and do to target ads or product recommendations to us, according to their patent applications filed with the U.S. government and reported by the New York Times.

Amazon and Google say their voice assistants currently only record and process audio after they’re triggered by a button or phrase like, “Hey, Alexa,” or “OK, Google.” Tech companies file many patent applications and sometimes nothing comes of them.

In one 2015 application, Google, which also owns smart-home device maker Nest, described a smart-home system that could be configured to identify emotions through “audio signatures of crying, laughing, elevated voices, etc.,” or send an alert when a child who was home alone got up to “mischief” based on whether the child was unusually quiet or active.

The tech giant—and world’s largest advertising platform—also outlined how a smart-home system with a camera could potentially recognize objects in one’s home, like a basketball or guitar, and serve up a related advertisement, in a separate 2015 patent filing. Another example detailed how an internet-connected device could recognize a t-shirt lying on the floor of a closet, identify the face on it to be that of actor Will Smith, check a browser’s search history to see if that person has searched for Smith recently, and then make a recommendation like,"You seem to like Will Smith. His new movie is playing in a theater near you."

The same filing talked about at how to secure that data by keeping it within a private network.

An Amazon patent filing described how a device could analyze a phone call between friends to understand what each person liked and disliked and serve them ads accordingly.

Amazon told the Times it did “not use customers’ voice recordings for targeted advertising.” And Google Home’s privacy policies say it doesn’t sell personal information to third parties.

“We file patent applications on a variety of ideas that our employees come up with,” Google also said in a statement. “Some of those ideas later mature into real products or services, some don’t. Prospective product announcements should not necessarily be inferred from our patent applications.”

Data privacy has exploded as a major public issue after reports last month that political-data firm Cambridge Analytica collected the data of 50 million Facebook users without their permission. Politicians in several countries have called for tighter regulation and consumer advocates have urged tech companies to be more transparent about how they gather and use personal information.

“These companies need to have the conversation with users about privacy and making such options clear, optional, and honest,” said Jason Herndon, director of technology innovation at RAIN, a digital agency that focuses on emerging technology like smart speakers. “People will tell you anything if they trust you, nothing if they don’t. What we can’t have is another Facebook scenario where it draws the line over what it can and can’t have access to without consent from the user.”