For enhanced search experiences, Google’s robots directive falls short. Google-Extended is a new standalone product token that website owners can use to tell Google not to use their site content for Bard, Vertex AI, and other AI projects, however it does not currently work for the AI-generated answers provided in the Search Generative Experience (SGE).
What is Google Extended?
Google extended was announce in March 2023 as a way to give website owners more control over how their content is uses by google (AI), it’s standalone product token that can added to a website’s robots.txt file, when a Google crawler sees the Google extended token, it will not use the content on that page for Bard, Vertex AI, or other AI projects.
What is the Search Generative Experience?
The search generative experience is a new feature that Google is testing, it provides users with AI-generated answers and summaries of search results, SGE answers are generated using a variety of factors, including the content of websites, the user’s query, and the user’s search history.
Why doesn’t Google Extended work for SGE?
Google has said that Google Extended does not currently work for SGE because SGE is a Search experiment. Website owners can still use robots.txt and the NOINDEX meta tag to manage their content in search results, including SGE.
However, it is important to note that blocking Googlebot from crawling your website will also prevent SGE from using your content, this means that if you want to prevent SGE from using your content, you will need to block Googlebot entirely.
Why is this important?
There are a few reasons, why this is important for website owners to know, First, some website owners may not want their content to be operated by the Google AI for any purpose, Second, some website owners may be concerned about the quality of the AI generated answers, summaries that are provide in SGE and third, some websites owner may be concerne about the privacy, data of their users which collected by Google AI.
What can website owners do?
Website owners have a few options if they want to prevent SGE from using their content. First, they can block Googlebot from crawling their websi and this can be done by adding the following line to their robots.txt file: User-agent: *
Disallow: /
Second, they can use the NOINDEX meta tag to prevent their content from being indexed in Search, and this can done by adding the following line to the HTML code of their pages:
<meta name=”robots” content=noindex>
Finally, they can use the Google-Extended Robots directive to prevent Google AI from using their content for Bard, Vertex AI, and other AI projects. This can be done by adding the following line to their robots.txt file:
Google-Extended: disallow /
It is important to note that blocking Googlebot from crawling your website will also prevent SGE from using your content. This means that if you want to prevent SGE from using your content, you will need to block Googlebot entirely.
Conclusion
The google extended robots directive, that is a new way for website owners to control how their content is used by the google AI. However. It does not currently work for the AI generated answers provided in the Search generative experience, website owners have a few options if they want to prevent SGE from using their content, but it is important to weigh the pros and cons of each option before making a decision.
Additional considerations
In addition to the above, here are a few other things to consider when deciding whether or not to prevent SGE from using your content:
- Impact on search traffic: If you block Googlebot from crawling your website, it will also prevent SGE from using your content and this could hurt your search traffic, as SGE answers are often featured prominently in search results.
- Quality of AI-generated answers: Google is constantly working to improve the quality of the AI-generated answers and summaries that are provide in SGE, the quality of these answers may improve over time, making them more useful to users.
- Privacy concerns: Google has said that it is committed to protecting the privacy of its users and the data that’s collected by google AI. However, some website owners may still be concerned about the privacy of their users if their content is used by SGE. Written by Abdul Bhai
