The Next Data Mine Is Your Bedroom

It’s a familiar feeling: Type something into Google’s search bar, and then start seeing ads for it everywhere. Sometimes, you don’t even need to search—Google’s already triangulated your desires based on your emails, your demographics, your location. Now, that familiarity stands to get a lot more intimate. With a fascinating pair of new patents for smart home technology, Google is hoping users will open their homes to its trademark eavesdropping.

In the first patent, Google imagines devices that would scan and analyze the surroundings of your home, then offer you content based on what they detect. According to the patent, the smart cameras in such a device could, for example, recognize Will Smith’s face on a T-shirt on the floor of a user’s closet. After matching this analysis against your browser history, the device might then say aloud “You seem to like Will Smith. His new movie is playing in a theater near you.”

It doesn’t stop at Will Smith movies. The patent imagines smart home devices would make all types of inferences about users, sorting them into categories based on what it sees in their most personal spaces. Using object recognition, it could calculate “fashion taste” by scanning your clothing and even estimate your income based on any “expensive mechanical and/or electronic devices” it detects. Audio signatures, too, could be used to not only identify users, but determine gender and age based on the timbre of their voices. The smart home would recommend what to watch and where to shop, all based on how it sorts users into categories of taste, income, and interest.

If this sounds invasive, it’s important to recognize this is already happening, just online. Google and Facebook both record and analyze user behavior, use it to sort people into categories, and then target them with ads and other content. Facebook likely knows your race and religion, while Google uses your emails and search history to sort you into ad-ready brackets. Netflix infers all types of data on users based on what they watch, then serves back hyper-specific movie and TV categories. This patent simply expands the areas in which your behavior is already mined and recorded from your phone and laptop to your bedroom.

And your children’s bedrooms. The second patent proposes a smart home system that would help run the household, using sensors and cameras to restrict kids’ behavior. Parents could program the device to note if it overhears “foul language” from children, scan internet usage for mature or objectionable content, or use “occupancy sensors” to determine if certain areas of the house are accessed while they’re gone— for example, the liquor cabinet. The system could be set to “change a smart lighting system color to red and flash the lights” as a warning to children or even power off lights and devices if they’re grounded.

While people can set goals for their children or themselves, these policies could also be “based upon certain inputs from remote vendors/facilitators/regulators/etc.,” according to the patent. That opens the door for companies to offer rewards for behaviors in the home. A household may set the internal goals of: “spend less time on electronic devices” or “use 5 percent less energy each month for the next 3 months.” Google devices could then connect to anything “smart” in the home and send you, and potentially a vendor or third party, updates on usage and screen time.

Just this month, the insurance company United Healthcare began partnering with employers to offer free Apple Watches to those who hit certain fitness goals. Insurers might also offer benefits to residents whose homes prove their fitness or brand loyalty—and punish those who don’t. Health insurers could use data from the kitchen as a proxy for eating habits, and adjust their rates accordingly. Landlords could use occupancy sensors to see who comes and goes, or watch for photo evidence of pets. Life-insurance companies could penalize smokers caught on camera. Online and in person, consumers are often asked to weigh privacy against convenience and personalization: A kickback on utilities or insurance payments may thumb the scales in Google’s favor.

For reward systems created by either users or companies to be possible, the devices would have to know what you’re doing at all times. The language of these patents make it clear that Google is acutely aware of the powers of inference it has already, even without cameras, by augmenting speakers to recognize the noises you make as you move around the house. The auditory inferences are startling: Google’s smart home system can infer “if a household member is working” from “an audio signature of keyboard clicking, a desk chair moving, and/or papers shuffling.” Google can make inferences on your mood based on whether it hears raised voices or crying, when you’re in the kitchen based on the sound of the fridge door opening, your dental hygiene based on “the sounds and/or images of teeth brushing.”

Of course, patents aren’t products, but they do represent an important shift. For a long time, the foundational metaphor of surveillance studies has been the panopticon—unending, inescapable, unwanted surveillance. Now, these patents seems to hint that the age of hyper-personalization will make people willing, enthusiastic participants in the panopticon, both as subjects and as architects.