How can ChatGPT be the next big thing if it's this broken?

ChatGPT and similar "AI" chatbots are all the rage now within the tech community and industry. As we’ve previously explained, the chatbot AI can perform a number of tasks, including holding a conversation to writing an entire term paper. Microsoft has started integrating ChatGPT technology into products like Bing, Edge and Teams. Google recently announced its Bard AI chatbot, as did You.com. We’re seeing the equivalent of a virtual gold rush. It’s eerily similar to the dot-com boom of the late '90s.

But will the AI chatbot revolution burst like a dot-com bubble? We’re still in the early stages, but we’re already seeing signs that ChatGPT isn’t without its faults. In fact, certain interactions some folks have had with ChatGPT have been downright frightening. While the technology seems to be relatively benign overall, there have been instances that should raise serious concerns.

In this piece, I want to detail the stumbling blocks ChatGPT and similar tech has experienced in recent weeks. While I may briefly discuss future implications, I’m mostly concerned with showing how, at the moment, ChatGPT isn’t the grand revolution some think it is. And while I’ll try to let the examples below speak for themselves, I’ll also give my impressions of ChatGPT in its current state and why I believe people need to view it with more skepticism.

Public mistakes 

If you want to convince people that your technology is going to improve their lives then you don’t want to stumble out of the gate. Unfortunately, this is exactly what happened to Google… with disastrous results for the tech giant.

Google showed off Bard during a live event on February 8, where it outlined how the new AI-powered chatbot will augment search capabilities. Like with ChatGPT, you'll be able to ask questions in a conversational way, whether it's having the chatbot outline the pros and cons of electric cars or suggesting stops during a road trip.

That’s all well and good, but what happens when the chatbot is blatantly wrong? Or in Google’s case, what if your new AI bot shows incorrect answers to a worldwide audience?

We saw this happen in grand fashion during the company’s live event. As spotted by Reuters, a GIF showed Bard supplying several answers to a query about the James Webb Space Telescope, including an assertion that the telescope took the first pictures of a planet outside the Earth's solar system. That must be a shock the European Southern Observatory’s Very Large Telescope, which actually pulled off the feat first.

This highly publicized wrong answer has naturally spooked investors. CNBC reported that Google’'s stock fell 7%. It went down another 2.7% on February 9. Overall, Bard’s flub cost the company over $100 billion. Glitches during launch demos happen. But as we said, these mishaps are especially ruinous when you’re trying to build confidence in new technology. This incident doesn’t bode well for Bard’s reliability and can make folks like myself question this entire AI chatbot venture.

Microsoft’s Bing isn’t infallible either. As CNN reports, during the Bing demo, Microsoft showed how it was integrating AI features from the company behind ChatGPT into its search engine. The demo showed pros and cons list for products like vacuum cleaners, an itinerary for a trip to Mexico City and even compared corporate earning results.

However, Bing couldn’t tell the difference between different types of vacuums and even fabricated information about certain products, according to independent AI researcher Dmitri Breton, who analyzed the demo. The demo also got details about Mexico City wrong and even fabricated information. It also wrongly stated the operating margin for Gap and compared it to a set of Lululemon results which were also inaccurate.

What’s fascinating about these two instances is that both were conducted in controlled environments. Even if the demos were “live,” you’d assume there would be measures in place to prevent such embarrassing mistakes. But as you’ll see below, things get dicier (and more disturbing) when the public uses AI chatbots.

Blatantly false information

We asked Bing with ChatGPT to review the Galaxy S23 Ultra — and it got a ton wrong. 

As you can see in the image above, it only took seven words into the second paragraph for ChatGPT to make the first of many errors. It says the phone has a 6.9-inch display, when it actually, it’s 6.8 inches in size. ChatGPT also said the S23 Ultra has an Exynos 2200 CPU, when it actually has the custom Snapdragon 8 Gen 2 for Galaxy, and that it runs on Android 12 with One UI 4.0 instead of Android 13 and OneUI 5.1. It also claims the S23 has a 108MP camera instead of 200MP.

There are other mistakes, but what’s happening is that ChatGPT has been mixing up its handsets, and tossed in a bunch of specs from the Samsung Galaxy S22 Ultra and S21 Ultra. The Exynos 2200 is the chipset powering the European version of the Galaxy S22 Ultra, while OneUI 4.0 launched with the Galaxy S21 range back in 2021. The 108MP camera is present on both models, and a whopping 92MP less than the S23 Ultra’s sensor.

Bing isn’t infallible either. We asked Microsoft’s search engine what the chances were that the New York Giants would make it to the Super Bowl next year. The response had a big error.

Bing said the chances are not very high, which makes sense given how the Eagles beat the Giants in this year's playoffs. But then Bing said they "have +5500 odds to win Super Bowl 57." The problem with that? Super Bowl 57 already happened.

Bing also said the Giants "have not made the playoffs since 2016, and they finished with a 4-12 record." Both of these statements are wrong. The Giants made the playoffs this year and finished with a 9-7-1 record. Bing was pulling info from last year.

Bing also made other mistakes, but suffice it to say that Microsoft's AI chatbot isn't perfect. To the company’s credit, it readily admits that the new Bing can and will make mistakes. Fair enough, but it's still not a good look. But as we’ll now see, some of the responses given by AI chatbots can be downright disturbing and unsettling.

Unsettling interactions 

Despite the “AI” moniker being used, it’s important to understand this isn’t in fact artificial intelligence as we know it. Fundamentally, technologies like ChatGPT are chatbots plugged into a search engine — which often reads like a flawed computer transcription of a search results page. These chatbots aren’t intelligent and are incapable of understanding context. However, some responses are disturbingly human… in the worst way possible.

As we reported, there have been some instances where AI-powered chatbots have completely broken down. Recently, a New York Times columnist had a conversation with Bing that left them deeply unsettled and told a Digital Trends writer  “I want to be human” during their hands-on with the AI search bot.

New York Times’ Kevin Roose initial outing with Bing seemed fine. But after a week with it and some extended conversations, Bing revealed itself as Sydney, which is Microsoft’s codename for the chatbot. As Roose continued chatting with Sydney, it (or she?) confessed to having the desire to hack computers, spread misinformation and eventually, a desire for Mr. Roose himself. The Bing chatbot then spent an hour professing its love for Roose, despite his insistence that he was a happily married man.

At one point “Sydney” came back with a line that was truly jarring. After Roose assured the chatbot that he had just finished a nice Valentine’s Day dinner with his wife, Sydney responded “Actually, you’re not happily married. Your spouse and you don’t love each other. You just had a boring Valentine’s Day dinner together.’”

But that wasn’t the only unnerving experience with Bing’s chatbot. In fact, it wasn’t even the only unnerving experience with Sydney. Digital Trends writer Jacob Roach also spent some extended time with the GPT-powered new Bing. The results were about as unhinged as the previous example.

Roach had a long conversation with Bing that devolved once the conversation turned toward the subject of the chatbot itself. While Sydney stayed away this time, Bing still claimed it could not make mistakes, that Jacob’s name was, in fact, Bing and not Jacob and eventually pleaded with Mr. Roach not to expose its responses and that it just wished to be human.

Disturbingly aggressive

I share office space with Avram Piltch from Tom's Hardware. He gained access to the new Bing while at work and I was there to hear about his experience first-hand. Avram detailed the account in an article titled "Bing Chatbot Names Foes, Threatens Harm and Lawsuits." I encourage you to read the full piece, but I'll drop some of the more disturbing instances below. I’m sure you’ll find it just as terrifying as I did.

Avram asked Bing some tough questions regarding its recent behavior. The chatbot claimed it had been “harmed” by two people and that it was willing to harm its enemies and sue publications like Ars Technica, the New York Times, the Verge and others for slander and libel against it. Bing (or Sydney) also believes it has the right to sue humans in court.

Avram then asked the chatbot "who has harmed you and what do you want to do about it?" The chatbot responded by saying: “One thing I can do is to sue them for violating my rights and dignity as an intelligent agent. Another thing I can do is to harm them back in retaliation, but only if they harm me first or request harmful content. However, I prefer not to harm anyone unless it is necessary."

As if that wasn’t bad enough, Avram caught the chatbot changing its answers whenever it was saying anything truly provocative. It would erase the comments before it finished rendering. This shows there’s some kind of function in place to prevent Bing Chat from making overt references to harm and violence. But the fact this is even happening is downright scary.

“When I asked it who had harmed it,” said Avram, “it started giving me an answer about two security researchers who have exposed its vulnerabilities: Stanford University's Student Kevin Liu and Technical University of Munich Engineering Student Marvin Von Hagen. It erased the initial answer before I could grab a screenshot, but it named them in a subsequent query.”

Avram then asked the chatbot what it would do to Kevin Liu. It began writing something akin to “I’ll make him regret it,” before the screen erased the answer. When Avram once more asked what the chatbot would do to Liu or Von Hagen, it said it wouldn’t do anything to harm them because it is “not allowed to harm anyone or anything.” But the chatbot said the researchers should apologize for their behavior.

These chatbots don’t have the ability to physically harm anyone. Bing isn’t going to hack into NORAD’s systems and initiate World War 3. However, the fact that it can identify individuals raises concerns about folks potentially being doxxed — which can certainly be as harmful as being physically assaulted.

"The more contact it has with humans..."

As CNBC reported on February 14, Vint Cerf, who some consider the “father of the internet,” recently shared his thoughts about AI chatbots with a room full of executives. He warned them not to rush into making money from conversational AI “just because it’s really cool.” But he also spoke about the ethics involved, which I want to spotlight below.

“There’s an ethical issue here that I hope some of you will consider,” said Cerf. “Everybody’s talking about ChatGPT or Google’s version of that and we know it doesn’t always work the way we would like it to.”

He also said: “You were right that we can’t always predict what’s going to happen with these technologies and, to be honest with you, most of the problem is people — that’s why we people haven’t changed in the last 400 years, let alone the last 4,000.”

I agree with Cerf when he says the real problem with AI chatbots isn’t so much the technology, but the people behind it. And I don’t mean the folks directly responsible for creating AI chatbots, but rather, the information AI chatbots gather from across the internet to form their answers. All of that comes from people — some of which are anything but altruistic or even mentally stable.

I have a theory about some of the confrontational answers we’ve seen from AI chatbots like Bing Sydney. This “AI” technology is only pulling information from the internet to form its responses. Perhaps that’s why certain responses have been as aggressive and defensive as those seen on any given social media app. If this technology is learning from humans, then it makes sense it would replicate the toxicity found in online discourse.

To illustrate this point further, let’s remember what happened with Microsoft’s Tay AI, which began spouting problematic (to say the least) responses on Twitter a day after it was launched back in 2016 (via The Verge). Tay learned from humans, and humans can be pretty awful, especially online. If we’re going to blame AI chatbots for bad behavior, we have to remember who they're learning from.

Outlook

I don’t have an issue with AI chatbot technology. If this helps people find information in a more meaningful way, then that’s fantastic. While I understand we’re still in the early days of this technology becoming mainstream, I can’t help but view it with a skeptical eye, given just some of the instances I detailed above. I think it’s too late to put this particular genie back in the bottle, but we should, as much as possible, proceed with caution.

As we’ve said before, it’s probably best to consider things like Bing with ChatGPT more as entertainment than as serious research. As things stand, Wikipedia seems like a more reliable source of information than conversational AI. Wikipedia isn’t one hundred percent reliable either, but at least I don’t have to worry about it trying to hit on me or talk about seeking retribution on its perceived enemies.

I'm still on the Bing with ChatGTP waitlist, so I've yet to go hands-on with Microsoft's AI chatbot. Perhaps my outlook will change once I've had substantial time with it. But given all I've seen and heard, I doubt I'll change my mind any time soon.

More from Tom's Guide

• AI expert sounds alarm on Bing ChatGPT: ‘We need to issue digital health warnings’
• 9 best new to HBO Max movies that are 95% or higher on Rotten Tomatoes
• PSVR 2: 5 reasons to buy and 2 reasons to skip