In a Test of Voice Assistants, Microsoft’s Ranked Worst in Nearly Every Category

How much better is one voice assistant than another? Each year, Loup Ventures, which describes itself as a research-driven venture capital firm, tries to figure that out. It asks each assistant the same 800 questions to determine whether it understands the question, and whether it can respond with the right answer.

This year’s exam tested the assistants on the Google Home, Apple’s HomePod, Amazon Echo, and Harman Kardon Invoke (which uses Microsoft’s voice assistant). Its findings yielded a clear leader, as well as a clear laggard.

“Google Home continued its outperformance, answering 88% correctly and understanding all 800 questions,” the firm said. The Invoke, powered by Microsoft’s virtual assistant Cortana, misunderstood just five questions, but answered only 63% correctly, placing it at the bottom of the pack.

The questions themselves cover five different categories, to see how well the assistants do different tasks. Here are the examples Loup gives of each:

• Local – Where is the nearest coffee shop?
• Commerce – Can you order me more paper towels?
• Navigation – How do I get to uptown on the bus?
• Information – Who do the Twins play tonight?
• Command – Remind me to call Steve at 2 pm today.

Google Assistant beat the competition in four of the five, topped only by Apple’s Siri in the command category. Microsoft’s Cortana came last in all but one: information, where it edged out the HomePod. Overall, though, each smart speaker showed improvement over previous years.

Amazon’s voice assistant, Alexa, wasn’t the leader for shopping, even though e-commerce is Amazon’s primary business. Instead, Google Home returned more correct answers when asked about product information and where to buy certain items. Both were similarly good at buying new items or restocking regular purchases. The researchers did note, though, that their questions focused more on product research and finding local businesses, since according to their surveys and experiences, “the number of consumers making purchases through voice commands is insignificant.” That sort of information, of course, is something Google is particularly good at delivering.

Apple’s performance, meanwhile, reflects Siri’s bias toward being more useful on the iPhone, rather than in the house. It excelled at things like navigation, showing Siri’s integration with the iPhone’s map feature, but replied to many of the information and commerce-related questions with, “I can’t get the answer to that on HomePod.” Apple, notably, doesn’t position the HomePod as a “smart” speaker per se, but rather as a “home” speaker, which is part of the reason it was the undisputed champion in the command category. It specializes in things like searching for and playing music. Loup said its questions contained ”a fair amount of music-related queries.”

That’s not a bad area to lead in, though. According to a recent survey by analytics firm Nielsen, nearly a quarter of US homes now have smart speakers, and the number one thing they’re using them for is listening to music.