Google-Extended and Googlebot control different uses of your content. In WordPress, you can block Google-Extended in robots.txt while leaving Googlebot allowed. That choice limits specified Gemini model training and grounding uses without removing your pages from Google Search. Blocking Googlebot is much broader: it can prevent Google from crawling pages needed for Search, including eligibility for AI Overviews and AI Mode.
Quick Answer
Keep Googlebot allowed if you want organic search visibility. Add a separate Google-Extended group only when your publishing policy calls for limiting Google’s non-Search generative uses. Then verify the public robots.txt file, test representative URLs, and document the decision. Do not use Google-Extended as an AI Overview opt-out because Google says Search AI features use Googlebot and ordinary Search controls.
Key Takeaways
- Googlebot supports Google Search crawling and affects Search features, while Google-Extended is a standalone robots.txt product token.
- Blocking Google-Extended does not block indexing or act as a Google Search ranking signal.
- Google-Extended has no separate HTTP user-agent string, so server logs cannot identify a distinct Google-Extended crawler visit.
- A specific Google-Extended group should state the policy you intend rather than relying on assumptions about a wildcard group.
- Robots.txt is public crawl guidance, not authentication, privacy protection, or a deindexing method.
Google-Extended vs Googlebot
| Question | Googlebot | Google-Extended |
|---|---|---|
| What is it? | A crawler token used for Google Search and related Search features | A standalone product token used as a control in robots.txt |
| Separate request identity? | Yes, Google documents Googlebot user-agent strings | No separate HTTP user-agent string |
| Search impact | Blocking can stop crawling needed for Search eligibility | Google says it does not affect Search inclusion or ranking |
| AI-related scope | Controls crawling for Google Search, including AI Overviews and AI Mode | Controls specified Gemini model training and grounding uses outside the Search control |
| Best default for an SEO site | Allow public, indexable content | Choose according to the publisher’s content-use policy |
Google’s current common crawler documentation provides the exact product scope and says Google-Extended does not affect inclusion or ranking in Search. Its Search AI guidance separately states that Googlebot is the robots.txt control for AI features built into Search.
Prerequisites Before Editing WordPress
- Access to the root robots.txt file through hosting, a trusted technical SEO tool, or the deployment repository.
- A copy of the current file, including every existing user-agent group and Sitemap line.
- A written decision: allow all Google-Extended uses, block them sitewide, or block only selected public paths.
- At least one public article URL and one intentionally blocked URL for testing.
- Google Search Console access for inspecting whether important Search pages remain crawlable.
If WordPress generates a virtual robots.txt file, confirm which plugin, host rule, or physical file controls the live response. Editing a dashboard field is not enough unless the public file at your domain root changes.
Step 1: Read the Live File First
Open https://yourdomain.com/robots.txt in a private browser window. Save the response before changing it. Confirm that it returns successfully, is plain text, and belongs to the correct hostname. A separate robots.txt file is required for each host and protocol scope that you operate.
Review the robots.txt guide if the file contains several groups or path rules. Google selects the most specific matching user-agent group, and groups targeting the same agent can be combined. This makes duplicate or conflicting groups harder to audit.
Step 2: Choose One Explicit Policy
To allow Google-Extended across the site, no special group is required when no applicable rule blocks it. For an explicit allow policy, use:
User-agent: Google-Extended
Allow: /To block the specified Google-Extended uses sitewide while preserving Search crawling, use:
User-agent: Google-Extended
Disallow: /
User-agent: Googlebot
Allow: /Keep your existing Sitemap line and any necessary rules for other crawlers. Do not paste this example over a working file. Merge the intended group into the current configuration, because WordPress sites often already exclude administrative or internal paths.
Step 3: Use Path Rules Only When They Match Policy
A selective policy can allow general content while blocking a defined directory:
User-agent: Google-Extended
Disallow: /licensed-reports/
Allow: /This is useful only when the directory boundary is stable and public URLs consistently live beneath it. A WordPress category archive does not necessarily contain the underlying post URLs, so blocking the archive path may leave individual articles allowed. Map actual URL paths before using a partial rule.
Google’s robots.txt documentation explains that the file belongs at the site root and that allowing is the default when no rule applies. It also documents group matching, which is why an explicit, consolidated Google-Extended group is easier to review.
Step 4: Verify Without Endangering Search
- Reload the public robots.txt URL and compare it with the intended text.
- Check that the Googlebot group was not changed to a sitewide disallow.
- Confirm that important CSS, JavaScript, images, and article paths remain available to Googlebot.
- Inspect a representative article in Search Console after Google processes the change.
- Record the date, rule, owner, reason, and URLs tested.
If your objective is controlling excerpts in Google Search, use supported snippet controls instead. The AI search snippet controls audit explains nosnippet, max-snippet, and data-nosnippet. Google states that pages need to be indexed and eligible for a snippet to appear as supporting links in AI Overviews or AI Mode.
Illustrative WordPress Example
Imagine a tutorial site wants public lessons in Search but does not want a licensed research directory used under Google-Extended. Its existing file permits normal crawling and declares a sitemap. The editor adds only a Google-Extended group for /licensed-reports/, preserves the Sitemap line, and leaves Googlebot unchanged.
| Check | Expected result | Priority |
|---|---|---|
| Homepage with Googlebot | Allowed | Critical |
| Public tutorial with Googlebot | Allowed | Critical |
| Licensed report path for Google-Extended | Disallowed | High |
| Public tutorial for Google-Extended | Allowed | High |
| Sitemap declaration | Unchanged and reachable | High |
This is an illustrative configuration, not evidence that the site will gain rankings or AI citations. Search inclusion still depends on indexing, eligibility, relevance, quality, and Google’s systems. See the Google indexing guide for the separate crawl-to-index process.
Troubleshooting Common Mistakes
Googlebot was blocked by accident
Restore the previous Googlebot rules immediately, confirm the live response, and inspect important URLs. A broad Googlebot disallow can affect Search crawling far beyond generative features.
The WordPress edit is not visible
Clear relevant caches and check whether a physical root file overrides a virtual WordPress response. Verify at the public URL, not only in the plugin interface.
Logs do not show Google-Extended
That is expected. Google says Google-Extended has no separate HTTP request user-agent string; existing Google agents perform crawling and the token acts as a usage control.
The rule did not remove a page from Search AI features
Google-Extended is not the control for AI Overviews or AI Mode. Review Googlebot access, indexing, and documented snippet controls. The article on Google AI Overviews explains the Search context.
Limitations and Maintenance
Robots.txt rules are public and voluntary for compliant crawlers. They do not secure confidential files, prevent copied content from circulating, guarantee deindexing, or prove how every AI system used a page. Google-Extended applies only to the Google uses Google documents for that token; it is not a universal AI opt-out.
Recheck the file after migrations, security changes, SEO plugin replacements, or CDN updates. Also revisit Google’s documentation because crawler scopes can change. A short quarterly audit is safer than accumulating overlapping groups that nobody owns.
Frequently Asked Questions
Does blocking Google-Extended block Google Search?
No. Google says Google-Extended does not affect a site’s inclusion or ranking in Google Search, provided Googlebot and other required Search crawlers remain allowed.
Does Google-Extended control AI Overviews and AI Mode?
No. Google identifies Googlebot as the robots.txt control for AI features in Search. Search snippet controls can limit how page content appears.
Can I find Google-Extended visits in server logs?
Not as a distinct user agent. Google-Extended has no separate HTTP request user-agent string and operates as a robots.txt control token.
Should I block Googlebot to stop AI training?
No. Blocking Googlebot can harm Search crawling. Use the narrowly documented Google-Extended token for its stated scope and preserve Googlebot access when Search visibility matters.
Will allowing Google-Extended improve rankings or AI citations?
No guarantee exists. Google says the token is not a Search ranking signal, and allowing it does not promise inclusion, citation, ranking, or traffic.
Conclusion
The safe WordPress setup starts by separating two decisions. Keep Googlebot accessible for pages you want in Search, including Search AI features. Configure Google-Extended independently according to your content-use policy, preserve the rest of the live file, and verify representative URLs. This narrow approach avoids turning an AI governance choice into an accidental organic search outage.