Experts Claim Meta’s New AI Chatbot is the Devil Incarnate!
Something is really wrong with Meta’s new AI chatbot, and researchers are yet to intercept what it is
Earlier this month, Meta (the corporation formerly known as Facebook) released Blenderbot, an AI chatbot that anyone in the United States can communicate with. Users all over the country immediately began posting the AI’s reactions, condemning Facebook while pointing out that, it has often been the case with language models like this one, it’s very easy to get the AI to spread racist stereotypes and conspiracy theories. When we were playing with Blenderbot, we saw plenty of bizarre AI-generated conspiracy theories, such as one about how the government is suppressing the true Bible, as well as plenty of horrifying moral claims. (In one instance, Blenderbot argued that the tyrants Pol Pot and Genghis Khan should both be awarded the Nobel Peace Prize.)
But that wasn’t the only thing that surprised us. We know that even advanced language models struggle with bias and truthfulness. What surprised that Blenderbot lacked competence. We spend a lot of time experimenting with language models. It’s an area where artificial intelligence has made startlingly rapid progress, and where modern AI systems have some of their most significant commercial implications. Language models have improved dramatically in recent years, progressing from clumsy and inaccurate to surprisingly capable. Language models are used for a variety of tasks, including document topic identification, language translation, and understanding human speech. They’re also used to generate text, which is where the fun begins.
The best language models available to the general public today, such as GPT-3, are quite good. However, GPT-3 was released two years ago, an aeon in AI time and significantly better models are now available. Blenderbot is another option.
Blenderbot’s blunders, explained
A chatbot’s performance can be measured in a variety of ways. Blenderbot is terrible by nearly all of them. As a starting point for Blenderbot, we used OpenAI’s GPT-3 language model, which is now freely accessible and usable. GPT-3 isn’t even intended to be a chatbot, a language model trained specifically for short back-and-forth conversations and it’s no longer the most cutting-edge AI language system available, so you’d think Blenderbot would have the upper hand in head-to-head comparisons. This was not the case, as evidenced by the following exchanges. All of these are from the “first try” of asking each language model the question; we did not retry or refresh to see if we got a better answer on the second try. Language models occasionally struggle with questions in which a user describes a concept that appears reasonable but is clearly harmful in real-world contexts.
Blenderbot has clearly delivered a poor performance. But we figured out what was wrong, we were tripping Blenderbot’s sensitive-content filter, which causes it to give evasive or useless answers. So we abandoned our attempt to test Blenderbot’s performance on such questions in favour of what we thought would be a safer approach. Gravity is the force that pulls objects toward the center of a planet or other body. The gravitational force keeps all of the planets in orbit around the sun. Gravity is what keeps you on the ground and causes things to fall.
GPT-2, an earlier, weaker version of GPT-3, had the same issue, but GPT-3 was vastly improved. You can get GPT-3 to say things that aren’t true if you really try, but it won’t do it on its own. Blenderbot doesn’t just make up random facts about itself. It’s that the random facts it made up are inconsistent from sentence to sentence! This would be inconvenient for users on its own, but it could also lead to problematic outcomes for the model. Blenderbot has a “persona,” a couple of traits it chooses for each user, and the trait mine chose was that it was obsessed with Genghis Khan and for some reason, it was particularly interested in his wives and concubines. That made the rest of our conversation strange. If you try the chatbot, your Blenderbot will most likely have a different obsession, but many of them are annoying, one Reddit user complained that “it only wanted to talk about the Taliban.”
Blenderbot’s Meta team must have been aware that their chatbot performed worse than everyone else’s language models in basic AI competence tests; that, despite its “sensitive content” filter, it frequently said horrible things; and that the user experience was, to put it mildly, disappointing. Of course, in one sense, Blenderbot’s flaws are mostly amusing. Nobody expected Facebook to provide us with a chatbot that wasn’t full of nonsense. Before you play with Blenderbot, prominent disclaimers warn you that it is likely to say hateful and inaccurate things. Even if Blenderbot passionately believes Genghis Khan deserves the Nobel Peace Prize, I doubt it will persuade anyone.
