The thing about AI chatbots, whether that’s Google Bard or ChatGPT, is they can rely on scraping the internet to answer your questions — and not everyone is on board with that. Presumably in a bid to stop complaints and potential lawsuits, Google is now giving web publishers the option to hide themselves from Bard’s gaze.
Google has announced that there’s a new control in its robots.txt indexing file called Google Extended. That file is how publishers can tell Google which URLs the company’s crawlers can access. The new control means publishers get to decide whether their content will "help improve Bard and Vertex AI generative APIs, including future generations of models that power those products."
According to Google’s announcement, it’s doing this as a result of feedback from web publishers about wanting to choose and control how content is utilized where generative AI is concerned. Though the language itself all focuses on how access means helping AI improve as time goes on. Almost as though Google is trying to suggest keeping Bard (and Vertex — it's cloud-based machine learning platform) off your work would be holding it back.
Generative AI has been a concern for some publishers, especially as companies like Google further integrate those services into products like Search. After all, every time a piece of information you create is used in a chatbot conversation, whether it’s cited or not, that means there’s one less person coming to your site for that same piece of information.
Fewer visitors would inevitably mean less revenue from advertising and whatever other revenue streams the site may rely on to keep operating. Of course other publishers may see other benefits to having their content appear in chatbot answers, and would be happy to have Bard crawling its content as a result.
The key thing there is about choice, because the last thing generative AI needs right now is more controversy. Plus when your service relies on being able to access and redistribute information as and when users ask for it, the last thing you want to do is upset the people providing it.
Though it does mean that, as a user, you may find that Google Bard isn’t going to be quite as adept as you’d like in the near future. Google will no doubt adapt to the loss of content from publishers who choose to opt out, and there’s so much content on the internet that you should still be able to get answers to your questions. But it will be interesting to see just how much of an impact this change could have on Bard’s knowledge going forward.
More from Tom's Guide
