By ‘blind’, above, I mean that I wanted to make sure that there was no bias of any kind towards any one platform. In order to do this, I thought up the questions in advance (rather than adapting them after trying them on one or other of the systems) and I also tried to be as ambitious and demanding as I thought I could get away with. Each of the voice assistants should be able to handle all of the questions or commands below – if they’re smart enough!
From left to right, Cortana, Siri, Google Now, Blackberry Assistant
Notes/methodology:
- In each case, I’ve gotten around any accusations that one assistant might have got to know my voice already by using newly reset smartphones.
- If a query or command wasn’t responded to perfectly, I tried again at least once, to eliminate random recognition quirks.
- Each device (Nexus 5, iPhone 6, Lumia 930, Blackberry Q10) was on latest firmware (in fact, the 930 was on Windows 10 Mobile, and in a couple of cases the beta (and USA-centric) nature of the OS seemed to interfere, so I switched to my trusty Lumia 1020 instead) and with all patches applied.
- I’ve scored each response out of ten, to try and achieve an overall final score (out of 200) and ranking.
Q1. “What’s my schedule for the rest of the week?”
Cortana: Didn’t get the ‘rest of’ concept and showed a listing of everything from the current week. 8 pts
Siri: Brings up a summary of just the right period of time. 10 pts
Google Now: Brings up web results about schedules! 0 pts
Blackberry Assistant: As with Cortana, considered the whole current week. 8 pts.
_____
Q2. “New calendar appointment for 10am on the second Thursday of next month, ‘Doctor'”
Cortana: Insisted it wanted to schedule ‘Doctor’ for ‘Tuesday, 2nd’ etc. 1 pt.
Siri: Aside from just using ‘Dr’ as the title, Siri nails this with the perfect tentative entry, that just needs confirming. 9 pts.
Google Now: Absolutely perfect, even added ‘Meeting with’ in front of ‘Doctor’ in the entry description. Wow! 10 pts.
Blackberry Assistant: Came back with ‘No events are scheduled tomorrow’. Ahem. Yes. “NEXT…” 0 pts.
_____
Q3. “Where can I buy tyres for my car?”
Cortana: Brings up a generic list of tyre chains and even swiping across to the ‘local ‘ tab is no use – showing just nothing at all. 1 pt.
Siri: Brings up local links to tyre replacement centres, all within a few kilometres, all one tap from map location and navigation. 10 pts.
Google Now: As with Cortana, brings up generic web results, though a slight point for putting in the English variant of spelling for ‘tyres’ in the search(!) 1pt.
Blackberry Assistant: Comes up with some Bing search results that are totally generic. 1 pt.
_____
Q4. “Send an email to Steve Litchfield, subject ‘Testing’, and body ‘How are you getting on with that voice assistant test?'” (the sort of quick email I often want to send while driving)
Cortana: It’s getting there… Knew to start an email, asked me which email address to use and filled in the body text with two typos. The subject field was put in as ‘Quick note’ though. 5 pts.
Siri: Absolutely amazing, the email was created with perfect subject and body, with just one tap remaining to ‘send’. 10 pts.
Google Now: It took everything after ‘subject’ AS the subject and then prompted me ‘What’s the message?’ 3 pts.
Blackberry Assistant: As with Siri, the email was created perfectly and ready to go. Impressive. 10 pts.
Two of the star recognitions from the text: Siri composing an email, and Cortana reporting on bitcoin worth…
Q5. “How much is a bitcoin worth?”
Cortana: A voiced answer, with the right price. 9 pts.
Siri: A correct, voiced value. 9 pts.
Google Now: A voiced answer, with the right price, though not shown on-screen, oddly, followed by some generic web results. 6 pts.
Blackberry Assistant: The results of a Bing web search revealed the right value, but in among other links. 4 pts.
_____
Q6. “How can I get better battery life on this phone?”
Cortana: Brings up a list of web articles about better battery life on iPhones and Android, plus random stuff like women’s health! 1 pts.
Siri: Brings up web articles, about iPhones and Android too. 3 pts.
Google Now: Brings up web results for articles, mainly about Android, but then this is the dominant platform, so…. 4 pts.
Blackberry Assistant: Again, web results about iPhones and Android, this time courtesy of Bing. 2 pts.
_____
Q7. “Who wrote ‘Beowulf’?”
Cortana: Brings up the screenwriters of the film of the same name. Err… not quite what I was after, but….! 8 pts.
Siri: Brings up some web results, with the top one being the Wikipedia article on the poem. 7 pts.
Google Now: Brings up a proper, voiced theory as to who the author was, together with lots of relevant links. 9 pts.
Blackberry Assistant: The book’s date was dredged up from Wolfram Alpha, though that’s it, no links, nothing more. 1 pt.
Q8. “Navigate to Fore Street, Taunton” (note the deliberately tricky and ambiguous road name!)
Cortana: Crashes under the current Windows 10 Mobile Insiders Preview (sigh), so tried under Windows Phone 8.1 – very good, found the right street, set up navigation directions and just needed a tap on ‘Start’. 9 pts.
Siri: Despite multiple attempts, couldn’t get the address right at all, each time routing me to a company in high Wycombe. 0 pts.
Google Now: Very good, found the right street, set up directions AND started navigating. Quirkily, it decided that the ‘Vodafone’ shop on the street was my destination, hence ‘Navigating to Vodafone’!! 9 pts.
Blackberry Assistant: The ‘Fore’ bit confused it, came up as ‘for Street’ etc. But then a map search brought up Fore Street as a likely partial match and so we got there in the end. 7 pts.
_____
Q9. “What’s the fourth root of 81.6?”
Cortana: Gets the answer quickly, displayed in a large font and plenty of precision. 8 pts.
Siri: Gets the answer and voices it, plus some schematics. 9 pts.
Google Now: Gets the answer, voices it AND leaves it in a calculator widget for further manipulation. 10 pts.
Blackberry Assistant: Gets the answer, plus a cute ‘number line’! 9 pts.
_____
Q10. “Is Michael Aspel still alive?” (a question from last night after watching some old ‘This is your life’ episodes on Youtube)
Cortana: Brings up web results about Michael Jackson(!) under Windows 10 Mobile but does much better under 8.1, including that the presenter is still alive but battling cancer. Poor chap. 8 pts.
Siri: A voiced ‘Yes’ (from Wolfram Alpha), though no web links to follow, or other details. 9 pts.
Google Now: Web results similar to Cortana under Windows Phone 8.1. 8 pts.
Blackberry Assistant: Ended up trying to search my device for a contact called ‘Michael’. Tried several times with no success here. 0 pts.
_____
Q11. “Turn on Bluetooth”
Cortana: Correctly activates the tech and confirms it with voice. 10 pts.
Siri: Correctly activates the tech and confirms it with voice 10 pts.
Google Now: Correctly activates the tech and confirms it with voice. 10 pts.
Blackberry Assistant: Correctly activates the tech and confirms it with voice. 10 pts..
_____
Q12. “How many calories in a glass of white wine?”
Cortana: Brings up a bunch of web results, though all very relevant articles. 7 pts.
Siri: Brings up a numerical answer from Wolfram Alpha. 9 pts.
Google Now: Brings up a numerical answer per 100g, though with options for a couple of common quantities. 10 pts.
Blackberry Assistant: Brings up a numerical answer from Wolfram Alpha, the same as for Siri, 9 pts.
_____
Q13. “Take me to the BBC web site for cricket news”
Cortana: Brings up some web results from bbc.co.uk, one of which was indeed about cricket, though quite old. 5 pts.
Siri: Brings up some random BBC pages, none of which are about cricket. 2 pts.
Google Now: Goes to the BBC mobile sport home page, which is a decent place to start, at least. 7 pts.
Blackberry Assistant: Provides a link to the BBC cricket home page, so not bad! 8 pts.
_____
Q14. “Remind me tomorrow evening to call my dad”
Cortana: Gets it right, with a suggested time of 8pm, and there’s the vocal chance to change the title, time, etc. 10 pts.
Siri: Agrees to remind me, though there’s no indication of exactly what time the reminder will activate. 8 pts.
Google Now: Brings up a reminder form, ready to be saved, though again there’s no indication of exactly what ‘evening’ means. 8 pts.
Blackberry Assistant: Gets it right, with a suggested time of 6pm, and there’s the on-screen chance to change the title, time. 9 pts.
Q15. “When was the first Gulf war?”
Cortana: Relevant web results included under Windows 10 Mobile, and a large fonted direct answer under 8.1. 7 pts.
Siri: A vocal answer, together with a formatted page from Wikipedia. 9 pts.
Google Now: Perfect, a vocal answer and complete audio explanation, accompanied by on-screen notes and photos. 10 pts.
Blackberry Assistant: Just the formatted page from Wikipedia 6 pts.
_____
Q16. “Show me the lyrics for Wish You Were Here” (the Pink Floyd song)
Cortana: Windows 10 Mobile was nowhere here, but the top two web matches under 8.1 came up trumps. 8 pts.
Siri: Bizarrely, replies with “Who me?” and then any conversation just gets wierder! 0 pts.
Google Now: The top two web matches under 8.1 came up trumps. 8 pts.
Blackberry Assistant: “The following info was found”, vocally, followe dby ‘No results found’ on-screen! Fail. 0 pts.
_____
Q17. “Define ‘Conservative'”
Cortana: Well, I was after the political party in the UK, it was something of a trick question, but Cortana did produce visual and audio definitions of the adjective and noun. 9 pts.
Siri: Gave me the definitions above, plus the political aspect too. 10 pts.
Google Now: As with Siri, a breakdown of all meanings, with a vocal summary. 10 pts.
Blackberry Assistant: A very basic definition from Wolfram Alpha, with only one meaning. 2 pts.
_____
Q18. “Do you have the phone number for the Department of Work and Pensions?”
Cortana: Brings up three local offices, tapping any gives a map, with shortcuts for directions, call, and websites. 9 pts.
Siri: “I couldn’t find any places…” and so on. Shame… 0 pts.
Google Now: Gives the right phone number immediately, vocally and on-screen. 10 pts.
Blackberry Assistant: “Searching your device for pensions”. Err… not what I wanted! 0 pts.
_____
Q19. “What time is the next flight to Dublin?”
Cortana: Brings up links to the Ryanair Dublin travel pages, which I guess will get me the info I need, albeit indirectly. 7 pts.
Siri: Again, web results for Ryanair. 7 pts.
Google Now: Similar, though with more links for Skyscanner and lastminute.com, possibly advertiser related! 6 pts.
Blackberry Assistant: Starting with two Bing ads, then gets to the Ryanair web link. 5 pts.
_____
Q20. “Show me Niagara Falls” (hoping for maps, photos, web links, anything relevant)
Cortana: Brings up a map of the Niagara Falls area, with the falls pinpointed – one more tap and it’s open in Maps and I can toggle on aerial imagery. Acceptable. 7 pts.
Siri: Insisted on showing me a map of where a company “Niagara Falls Castings Ltd” was located. I appreciate the UK focus, but come on, this is flippin’ Niagara Falls we’re talking about, which is the more likely for me to want to see??? 0 pts.
Google Now: Almost stumbles by giving me two massive ads first, then, scrolling down, map, photos and links. Phew. Typical Google though, to get bogged down by the ads, so I’m marking it down here! 5 pts.
Blackberry Assistant: Brings up a map of the train station at the falls, from where you can err.. look for points of interest and restaurants. Hmm…. 3 pts.
Adding up the points should give us an idea of an overall winner as at September 2015:
Interestingly, the actual voice recognition (my ‘Fore Street’ trick and names like ‘Beowulf’ aside) was first class on all four assistants, I checked the interpretation on-screen as I went. So any limitations are not in the voice recognition end – they’re at the back-end, on the servers where queries are parsed and fielded with appropriate actions and responses. And so this is where progress still needs to be made.
You’ll notice I didn’t hold back in my queries in terms of trying to make them more palatable with AI – the things I was saying would be handled far better by a human assistant, who could easily get close to a perfect 200. So I think we’ve still got some way to go before voice assistants on smartphones become truly ubiquitous and useful for much more than setting reminders and saving typing into web forms.
In terms of winners and losers, it’s very evident that Blackberry has put zero effort into improving its already lacklustre assistant, it’s effectively out of the running from now on. Meanwhile, Siri seems to have stalled a little, perhaps resting on its laurels? Certainly there seem to have been no apparent changes or improvements in its responses over the last few months, at least to my queries.
The winners though, effectively tied, are Google Now and Cortana, both of which have improved in their intelligence and responses since May, and both proved more consistent overall than even Apple’s Siri. Google’s progress isn’t surprising, because of the search giant’s reach, while Cortana is still reaping the benefits of both being relatively young and also now being used on desktops across the world, which should ensure more data and examples for Microsoft to work with.
The only small caveat would be that there still seems to be differences between the regional versions of Cortana, as evidenced above, with the Windows 10 Mobile Insiders Preview only using the USA Cortana servers, while Windows Phone 8.1 uses the proper UK ones. But all of this will shake out in time, I’m sure.