Quick answer: Applebot crawls and renders webpages for Apple search experiences such as Spotlight, Siri, and Safari. Applebot-Extended does not crawl pages; it is a robots.txt control that lets publishers opt out of having Applebot-crawled content used to train Apple’s general-purpose foundation models. A WordPress publisher can allow Applebot for search while separately allowing or blocking Applebot-Extended.
Applebot vs Applebot-Extended at a glance
| Control | Primary role | What blocking changes |
|---|---|---|
| Applebot | Crawls and may render content for Apple search features | Stops affected URLs from being crawled by Applebot |
| Applebot-Extended | Controls use of Applebot-crawled data for model training | Opts affected content out of foundation-model training |
| nosnippet | Controls descriptions and web answers | Prevents Applebot from generating a description or web answer and using tagged data as live AI context |
| noindex | Controls search eligibility | Prevents the page from appearing in Spotlight or Siri Suggestions |
These controls are not interchangeable. Apple states that Applebot-Extended does not crawl webpages and that blocking it does not remove a page from Apple search results. Apple also says Applebot-Extended rules are not considered when ranking Search. Review the current Applebot documentation before changing a live policy.
Prerequisites for the WordPress setup
- Access to the public robots.txt file at the exact HTTPS hostname.
- Permission to change the WordPress SEO plugin, host, server, or CDN that generates robots.txt.
- A written choice for search crawling, model training, and live AI answer context.
- One published test URL and access to its rendered HTML or response headers.
- Optional server or CDN logs for confirming genuine Applebot requests.
Take a copy of the current file before editing. On WordPress, robots.txt may be physical, virtual, or rewritten by an SEO plugin. Change only the system that actually serves the public file. The general robots.txt guide explains file location and crawl rules.
Choose an Apple crawler policy first
Decide what the site intends before writing directives. A public educational site may allow Applebot for discovery and make an independent decision about model training. A private member area should normally remain inaccessible to public crawlers regardless of AI policy.
| Goal | Applebot | Applebot-Extended | Extra control |
|---|---|---|---|
| Allow search and training | Allow | Allow | Use normal index and snippet settings |
| Allow search, opt out of training | Allow | Disallow | Keep snippets unless live AI context is also unwanted |
| Keep a section out of Apple search | Disallow or noindex as appropriate | Match the content policy | Protect private content with authentication |
Blocking a crawler is not access control. Sensitive material must require authentication because robots.txt is public and compliant crawlers are free to follow it voluntarily.
Step 1: audit the current robots.txt groups
Open the live robots.txt file in a private browser window. Confirm that it returns plain text with HTTP 200. Search for Applebot, Applebot-Extended, Googlebot, and the wildcard group.
Apple’s documentation contains an unusual fallback: when robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot instructions. An inherited Googlebot block can therefore affect Apple discovery even when Applebot is never named. Applebot does not support crawl-delay.
Google’s robots.txt documentation confirms that rules belong in groups beginning with a user-agent and that the file must live at the host root. Use plain UTF-8 text and avoid curly quotes or copied formatting.
Step 2: add the narrow policy you selected
To allow Apple search crawling while opting out of foundation-model training across the public site, use separate groups:
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Disallow: /If training is allowed, omit the Applebot-Extended block instead of adding an empty or contradictory group. If only one directory should be excluded from training, disallow that path rather than the entire site:
User-agent: Applebot-Extended
Disallow: /members-only-guides/Preserve existing sitemap declarations and necessary rules for admin, cart, search, or account URLs. Do not copy a sample path unless it exists on the site.
Step 3: separate training from live AI answer controls
Applebot-Extended addresses model training. It does not control every use of current page content in generated output. Apple documents the nosnippet directive for pages that should not produce a description or web answer or be used as additional live context in Apple AI output.
A page-level Apple-specific directive can be added in the HTML head:
<meta name="applebot" content="nosnippet">WordPress themes and SEO plugins vary, so confirm the rendered source rather than trusting a settings screen. Do not combine nosnippet with noindex unless the page should also disappear from relevant Apple search experiences. The AI search snippet-controls audit provides a broader verification workflow.
Step 4: test crawling and rendering
- Reload the public robots.txt file and confirm the exact group and path appear once.
- Fetch a representative published page and verify HTTP 200, a self-referencing canonical, and the intended robots meta directive.
- Confirm CSS, JavaScript, images, and XHR resources needed for the main content are not accidentally blocked.
- Clear the relevant WordPress, host, and CDN caches, then repeat the public checks.
- Record the policy, test URL, date, and final result for the next audit.
Apple says Applebot may render pages in a browser and may fail to render properly when required resources are blocked. A readable server-rendered page or graceful degradation reduces reliance on client-side execution. Review the site’s JavaScript SEO guide if meaningful content appears only after scripts run.
Step 5: verify genuine Applebot activity
Filter server or CDN logs for user agents containing Applebot. Record the timestamp, requested URL, response status, and source IP. A user-agent string alone is not proof.
Apple recommends reverse DNS verification in the applebot.apple.com domain or matching the IP against its published Applebot CIDR JSON file. Applebot-Extended should not appear as a crawler request because Apple says it does not crawl. Investigate any request claiming that identity rather than allowlisting it.
Worked example: a training opt-out without losing search
Consider an illustrative WordPress tutorial site. The owner wants articles discoverable through Apple search but does not want them used for foundation-model training. The current robots.txt has only a wildcard group, and the articles are indexable.
- The publisher adds an Applebot group that allows public paths.
- A separate Applebot-Extended group disallows the entire site.
- No nosnippet tag is added because search descriptions and live answer context remain acceptable.
- The public robots.txt and two articles are retested after cache clearing.
- Logs are monitored for genuine Applebot responses, while no Applebot-Extended request is expected.
The result expresses two separate choices: retain Apple search crawling and opt out of model training. It does not promise rankings, Siri links, Apple Intelligence citations, or referral traffic.
Troubleshooting common mistakes
- Applebot remains blocked: inspect Googlebot fallback rules, wildcard rules, CDN blocks, and the final public file.
- The edited file does not change: identify whether WordPress, an SEO plugin, the host, or a physical root file controls the response.
- A page still appears in search: Applebot-Extended is not a noindex control; review Applebot crawling and page-level indexing directives.
- The page is crawled but renders poorly: check blocked scripts, styles, XHR requests, and server-rendered content.
- Logs show Applebot-Extended: treat the claim as suspicious because that control token does not crawl webpages.
Limitations and measurement
Robots.txt confirms policy, not compliance evidence for every past use. A log entry confirms a request, not indexing or an AI citation. Analytics can identify referred visits when a source is present, but it cannot measure unseen mentions or complete Apple AI exposure. Do not claim that allowing Applebot, adding schema, publishing llms.txt, or permitting model training guarantees inclusion. Measure observable crawling, indexability, referrals, and business outcomes separately.
Conclusion
The correct Applebot versus Applebot-Extended WordPress setup begins with three independent decisions: search crawling, foundation-model training, and live AI answer context. Apply the narrowest matching robots and page directives, test the public output, and verify genuine Applebot activity. Clear separation prevents a training preference from accidentally becoming a search visibility problem.
Frequently Asked Questions
Does Applebot-Extended crawl webpages?
No. Apple says Applebot-Extended does not crawl webpages. It controls how data already crawled by Applebot may be used for foundation-model training.
Can I block Applebot-Extended and remain in Apple search?
Yes. Apple states that pages disallowing Applebot-Extended can still appear in search results when Applebot and the page’s indexing rules allow them.
What does nosnippet do for Applebot?
It prevents Applebot from generating a description or web answer and stops tagged data from being used as additional live context in Apple AI output.
Does Applebot follow Googlebot rules?
Apple says Applebot follows Googlebot instructions when robots.txt does not mention Applebot but does include a Googlebot group.
Will allowing Applebot guarantee Apple Intelligence citations?
No. Allowing crawling only removes one technical barrier. Apple decides whether content is indexed, ranked, linked, or used in a generated response.