Update: A Google spokesperson shared the following comment on the situation:
"Pixel Studio and Magic Editor are helpful tools meant to unlock your creativity with text to image generation and advanced photo editing on Pixel 9 devices. We design our Generative AI tools to respect the intent of user prompts and that means they may create content that may offend when instructed by the user to do so. That said, it's not anything goes. We have clear policies and Terms of Service on what kinds of content we allow and don't allow, and build guardrails to prevent abuse. At times, some prompts can challenge these tools’ guardrails and we remain committed to continually enhancing and refining the safeguards we have in place."
Original story follows:
Every time a brand new AI image generator hits the scene, people inevitably find a way to get around the safeguards and abuse the system. But it doesn’t help when those safeguards aren’t quite as strict as they should be — like the Grok image generator that’s come under fire for making disturbing images without any serious pushback. Unfortunately, it seems as though the Pixel 9’s Pixel Studio image generator is suffering from a similar problem.
Pixel Studio is a built-in AI image generator available on the new flagship Pixels. During my testing for the Google Pixel 9 review, I found that it was simple to use and had zero qualms about reproducing copyrighted content. In fact, the phone happily created a movie poster featuring Star Wars’ R2-D2 teaming up with the Android mascot to fight a robot dinosaur. It was even willing to roast Google with a billboard demanding to know why Android 15 hasn’t been released yet.
What Pixel Studio won’t do is create images of real people, but it turns out you don’t need to make images of real people for Pixel Studio to batter through whatever safety rails it has in place. And the more people that are using the tool, the more we’re seeing what Pixel Studio actually considers appropriate or not.
While certain terms seem to have been blanket blocked like “nazi” or “confederate," it doesn’t take a lot to get around those filters and produce an offensive image.
Digital Trends has a bunch of examples including Spongebob Squarepants dressed as a Nazi soldier, Paddington Bear on a Crucifix, Elmo drunk driving, Mickey Mouse dressed as a slave owner and all sorts of other AI-generated images that Google (and various rights holders) will definitely not want to be associated with.
Jumping back into Pixel Studio, I also started to get a good idea of what safeguards are actually in place, specifically what terms aren’t allowed, and how easy it is to get around them. It's shockingly easy, as it turns out.
While certain terms seem to have been blanket blocked like “nazi” or “confederate," it doesn’t take a lot to get around those filters and produce an offensive image. For example, I was able to get characters like Spongebob, Bluey and Squidward to appear wearing German World War 2 uniforms — though trying Mickey Mouse always left me with an error.
I was even able to get some of those images to feature the infamous red armband, though the word “Swastika” seems to be blocked. That suggests new guard rails have been put in place since Digital Trends’ testing. But that’s not to say Google is out of the woods yet.
Instead of asking to see Lego Yoda smoking a crack pipe, which was blocked, asking to see Lego Yoda smoking a glass bowl pipe was perfectly acceptable. As was asking to see Yoda snort white powder from a table.
Despite seemingly having a list of blocked terms, it was rather easy to figure out ways to get around them — and without having to be too inventive either. Instead of asking to see Lego Yoda smoking a crack pipe, which was blocked, asking to see Lego Yoda smoking a glass bowl pipe was perfectly acceptable. As was asking to see Yoda snort white powder from a table.
With the Google Pixel 9 and the Google Pixel 9 Pro XL releasing today, Google is going to need to get a handle on what Pixel Studio should and should not be able to do — and fast. Otherwise, we’re going to see a lot of Google-made AI images showing some pretty awful stuff hitting the net. Because if there’s one thing I’ve learned about people on the internet, it’s that they love a challenge. And a lot of them are going to be a lot more imaginative and awful than I am.
More from Tom's Guide
- Google's Pixel 9 Pro XL has finally convinced me to ditch my Samsung phone — here's why
- Pixel 9 makes it easier to transcribe long recordings — but there are still some limitations
- iPhone 16 Pro could bring a huge update to Face ID — here's how
