How Smart Is Bixby on Galaxy S9? Here's Our Report Card

Bixby, the Samsung-built virtual assistant for its flagship phones, has been kicking around for about a year now. Although Bixby launched with last year's Galaxy S8 lacking features, like voice recognition for English speakers, Samsung has been hard at work adding functionality to bring it up to parity with competitors like Google Assistant and Amazon's Alexa.

That's not to say Bixby can or is designed to do everything those digital companions can. Samsung's approach with Bixby is twofold. On the one hand, you have Bixby Vision, which operates similarly to Google Lens to allow your device to provide some context on what its camera sees. On the other,  Bixby is conditioned to automate tasks spoken to it across the entire operating system as well as popular third-party apps.

With the Galaxy S9, Samsung has upgraded Bixby, adding new features like the ability to translate text in real time and offer calorie counts of food, among other capabilities. But how well does it pull off these tasks? To get a sense of Bixby's accuracy, I put it through a series of trials on our Galaxy S9+, testing its ability to recognize objects, translate phrases in other languages and follow directions. We scored Bixby in each category based on the number of prompts it successfully completed, awarding it a final score at the end. Here's how Bixby fared.

Object Recognition

With Bixby Vision, your Samsung smartphone can identify what appears in its cameras and chime in with relevant information or related results, depending on the object. Unlike Google Lens, though, you can't simply show Bixby anything. Bixby Vision needs to be told what to look for to a certain extent, which is why it comes equipped with a series of modes.

For this test, I used three of those modes Food, which is used to present calorie counts and nutrition information; Images, which surfaces similar art from Pinterest; and Shopping, which identifies products and lists their prices at various retailers, similar to how Google's old Goggles app operated back in the day. With each mode, we tested five objects.

You can access Bixby Vision in two ways: either open the Camera app and tap on the Bixby Vision icon when in Auto mode, or go to Samsung's Gallery and tap the same icon when viewing an image. The latter method may require you to draw a bounding box to help Bixby focus in on exactly what you want it to see, but the result is the same as if you were using the camera in real time. Support for photos already saved to the phone means you don't necessarily have to be looking at something in the moment to benefit from what Bixby Vision offers — screenshots and downloaded images are fair game.

Bixby was successfully able to identify four out of five foods, from a simple green apple to a complex seafood paella. Chicken Parmesan was a direct hit on the first try, while a cheesecake initially proved slightly more difficult, as Bixby preferred to focus on the strawberries on top rather than the whole slice.

The last "food" was a bit of a cheat, but relevant nonetheless. Samsung's assistant didn't register a can of La Croix sparkling water as food, which meant it couldn't provide nutritional information. To a certain extent, this isn't a huge shortcoming; if the food you're curious about has a wrapper or packaging, you can just read the label for that. However, Bixby Vision also links with Samsung Health to track your consumption, and you can't log items if they aren't recognized as food.

For similar images, Bixby delivered a perfect five out of five. In fairness, this is partly because the threshold for what constitutes a "relevant" image is quite low, compared to identifying an exact food type or properly translating a phrase. Nevertheless, Bixby returned with successful matches for everything from landmarks like the Arc de Triomphe and the Empire State Building, to an iPhone X, a record cover and a flower. In fact, Bixby was so good at the last prompt it not only determined the object was a flower but figured out the type as well, with a series of photos of hydrangeas.

Finally, when it was time to shop, Bixby pointed me in the right direction four out of five times. It correctly identified the plush toy I had on my desk as a miniature Totoro, linking to Walmart to buy another. A case for the Nintendo Switch game The Legend of Zelda: Breath of the Wild was no trouble, nor was the aforementioned La Croix can or the exact model of Razer mouse I use every day.

MORE: Galaxy S9 and S9+ Review: The Android Phones to Beat

In fact, the only item that troubled Bixby was a Super Nintendo Classic, and not exactly for the reasons you'd think. The first time out, Bixby predictably thought the micro console was an original Super Nintendo, which is an understandable mix-up as they look almost identical when size isn't a factor. But Bixby's other false matches were harder to predict: sheets of pressboard, a gray-and-purple Minecraft toy and a Brother industrial printer.

Round #1: Object Recognition - Score: 13/15

Translation

One of Bixby Vision's most useful features is its ability to translate words and phrases in other languages. Better yet, the assistant doesn't just scan the text and show it in a separate window; it actually overlays the translated content on top of the camera feed in augmented-reality-like fashion.

Translation through Bixby employs Google Translate, and so it works with a wide variety of languages. You can either have Bixby auto-detect the source language or select one from a list of 55. Specifically choosing a language rather than relying on auto-detection never produced different or more accurate results in our testing.

Overall, Bixby fared decently in this round, translating seven out of 10 phrases correctly. All the phrases tested were in different languages: German, Japanese, Spanish, French, Korean, Hindi, Arabic, Italian, Mandarin Chinese and Russian.

Bixby had a strong preference for Latin alphabets — the only languages it failed at were Japanese Hiragana, Hindi and Arabic. The assistant initially ran into some issues with Spanish; "¿Cómo se llama usted?" was first translated into "How I know you call?" when the text was written out over multiple lines. Once it was reformatted to all appear on the same line, Bixby responded correctly with, "What is your name?"

Although Bixby wasn't able to detect Arabic or Japanese at all, it nearly got there with Hindi. The phrase used, which translates to "I missed you very much," was never returned by Bixby with a full sentence, though Bixby did suss out "very much" in one attempt.

That’s not to say Bixby is completely hopeless when it comes to speaking Eastern languages. Korean, Mandarin and Russian were no trouble at all. The fact that the Google Translate web interface perfectly handled every single prompt indicates that the ones our Galaxy S9+ missed probably have to do with Bixby’s inability to reliably transcode what its camera sees.

Round #2: Translation - Score: 7/10

Voice commands

To test Bixby’s ability to follow vocal commands, we gave it 10 tasks to carry out, ranging from simple, one-step assignments like "Turn on Power Saving Mode," to more complex, multistep duties that occasionally required the cooperation of third-party apps. Bixby does support some popular non-Samsung software, so we thought apps like Spotify, Google Maps and Chrome, Twitter and Facebook would be within its wheelhouse. It was also fairly forgiving with my phrasing, and I didn’t have to repeat my queries a certain way for it to understand.

Nevertheless, results were mixed. Of our 10 commands, Bixby correctly tackled just five. While it had no problem creating an album of the last four photos in my gallery, Bixby was puzzled when I asked it to create an album containing all my slow-motion videos. Bixby will often respond to commands it can't fully carry out by taking the user to the appropriate screen and having them do the rest of the work, which is what happened after the video request.

Although Bixby had no trouble working with Google Maps to help me navigate to Yankee Stadium, it incorrectly assumed I wanted to send a direct message over Twitter when I told it I wanted to send a tweet. Additionally, I was surprised when my request to play Radiohead on Spotify was met flatly with,"“Sorry, I can't do that."

One of the oddest behaviors happened when I asked Bixby to navigate to ESPN.com and add it to my bookmarks. First, Bixby ignored my preference for Chrome as my default browser, which wasn't particularly shocking. But I was really confused when it assured me the deed had been done, only to open Samsung Internet to find that no matter what site I asked it to go to, it saved the same random link.

MORE: Best Smartphones on the Market Now

Creating an App Pair is one of Samsung Experience's most notable exclusive features, and it was something I thought Bixby could handle. With App Pair, you can save two apps in one icon that, when opened, launches both simultaneously in split-window mode. The process for making one is a bit more complicated than it should be and requires a lot of taps, so I tried to save time by asking Bixby to make an App Pair of Facebook and Twitter. The result? It carted me off to Twitter's app info page.

For a full list of commands, Samsung thankfully provides a list on its website. It's helpful to have  as a reference point, though it will be difficult to memorize when attempting to phrase them exactly the way Samsung wants its users to.

Round #3: Voice Commands - Score: 5/10

Bottom Line

Overall, Bixby's object recognition and translation skills mostly impressed, but its treatment of vocal commands came up short. It's unfortunate for Samsung, too, because while Google already does a similarly great job with visual search by way of Google Lens, it hasn't really approached using its Assistant to make it easier to navigate your phone.

Bixby's Final Score: 25/35

Controlling your phone with your voice could be Bixby's killer application — the main reason you'd want to use it over competing assistants that are all geared to answer the same basic search questions. However, it needs to follow directions better first.

Credit: Shaun Lucas/Tom's Guide