Quick answer: An Amazon crawler policy for WordPress should treat Amazonbot, Amzn-SearchBot, and Amzn-User separately. Amazonbot may support product improvement and AI model training; Amzn-SearchBot supports Amazon search experiences without generative AI training; and Amzn-User fetches live information for user-initiated actions. Audit your current robots.txt file, choose a purpose-based policy, publish explicit rules, and verify real requests with Amazon’s official IP lists.
Key Takeaways
- Amazon documents three independent user agents with different purposes.
- Blocking Amazonbot does not automatically block Amzn-SearchBot or Amzn-User.
- Amzn-User may not follow every robots.txt directive because a person can trigger its fetch.
- A user-agent string alone does not prove that a request came from Amazon.
- Robots.txt manages crawler access; it is not a security control or a guarantee of visibility.
What the Three Amazon Crawlers Do
The names are similar, but the policy decision is not. Amazon’s official crawler documentation says each setting is independent and may take about 24 hours to be reflected.
| Agent | Documented purpose | Important control detail |
|---|---|---|
| Amazonbot | Improves Amazon products and services and may train Amazon AI models | Use a dedicated robots.txt group when you want an explicit training-related choice |
| Amzn-SearchBot | Improves search experiences in Amazon products, including eligibility for experiences such as Alexa | Amazon says it does not crawl for generative AI model training |
| Amzn-User | Fetches current web information for user actions, such as an Alexa query | It may not follow all robots.txt directives because a user can initiate the request |
This separation is similar to the purpose-based choices explained in the OAI-SearchBot versus GPTBot guide, but the names and documented behavior are platform-specific. Never copy one company’s crawler rules into another company’s configuration.
Prerequisites for an Amazonbot WordPress Setup
- Access to your live robots.txt file at your domain followed by /robots.txt.
- A backup of the current file or the plugin setting that generates it.
- Permission to edit hosting files, an SEO plugin’s robots editor, or WordPress code.
- Access to server, CDN, security, or hosting logs if you plan to verify visits.
- A written decision about search visibility, AI training, and user-triggered retrieval.
Read the robots.txt beginner guide first if user-agent groups and Allow or Disallow rules are unfamiliar.
Step 1: Audit the Current WordPress Robots.txt
Open the live file in a private browser window. Check whether it is a physical file in the web root, a virtual WordPress response, or output generated by an SEO or security plugin. Use only one editing method so two systems do not overwrite each other.
- Save the complete current response.
- Look for groups naming Amazonbot, Amzn-SearchBot, Amzn-User, or a wildcard agent.
- Check for a sitewide Disallow rule and confirm that WordPress search visibility is enabled.
- Review CDN or firewall rules that may block bots before WordPress receives the request.
- Confirm that important public pages return successful responses without login or cookie requirements.
WordPress exposes a robots_txt filter for virtual output. If your host serves a physical file first, that filter will not control the public response.
Step 2: Choose a Purpose-Based Policy
| Goal | Amazonbot | Amzn-SearchBot | Amzn-User |
|---|---|---|---|
| Allow all documented uses | Allow | Allow | Allow, while recognizing its special behavior |
| Decline model-training crawl but permit search eligibility | Disallow | Allow | Decide separately |
| Reduce all optional Amazon crawling | Disallow | Disallow | Do not treat robots.txt as enforcement |
The second policy is often the most useful example because it separates training from search. It does not promise appearance in Alexa or any AI answer. Eligibility still depends on Amazon’s systems, the usefulness of the page, and other technical signals.
Step 3: Add Explicit Robots.txt Groups
For a site that declines Amazonbot while permitting Amzn-SearchBot, the core rules are:
User-agent: Amazonbot
Disallow: /
User-agent: Amzn-SearchBot
Allow: /You can add an explicit Amzn-User group, but Amazon warns that it may not follow every robots.txt directive. Do not place confidential, licensed, customer, staging, or account data on a public URL and depend on a bot rule to protect it. Use authentication or access controls for private material.
Avoid adding Crawl-delay. Amazon states that these agents do not support it. Keep existing rules for Googlebot and other services unchanged unless a separate audit justifies changes.
Step 4: Verify the Live File
- Reload the exact public robots.txt URL and confirm the intended groups appear once.
- Test both the www and non-www host if both resolve, because robots.txt is host-specific.
- Check capitalization, blank lines, colons, and path prefixes.
- Confirm that the sitemap line and unrelated crawler groups remain intact.
- Wait at least 24 hours before judging whether Amazon has reflected a changed preference.
Do not use robots.txt to remove a WordPress page from search results. Google explains that robots.txt controls crawling and is not a reliable way to prevent indexing. Use a suitable noindex method or authentication for that separate goal.
Step 5: Verify Amazon Requests in Logs
Filter logs by the exact agent names, then validate source addresses against Amazon’s published lists for Amazonbot, Amzn-SearchBot, and Amzn-User. A copied user-agent string can be spoofed.
Record the timestamp, requested URL, response code, user agent, and verified source range. The workflow in the crawler log verification guide shows how to separate claimed bots from validated requests without treating every visit as AI visibility.
Worked Example: A Publisher Allows Search but Declines Training
Consider an educational WordPress site that wants its public tutorials eligible for Amazon search experiences but does not want Amazonbot used for model training. The owner keeps the public articles crawlable, blocks Amazonbot at the root, explicitly allows Amzn-SearchBot, and leaves private course material behind authentication.
After publishing the file, the owner confirms the live response, checks that Googlebot rules were not changed, and reviews logs a week later. An Amzn-SearchBot request with a successful response and an address inside Amazon’s published range confirms access. It does not prove indexing, an Alexa answer, a citation, or future traffic.
Troubleshooting Checklist
| Problem | Likely cause | Next check |
|---|---|---|
| Rules disappear | A plugin, cache, or physical file overrides WordPress output | Identify which layer serves the live response |
| Expected crawling does not appear | No demand, delayed refresh, blocked resources, or firewall filtering | Check the live file, response codes, CDN logs, and Amazon IP ranges |
| Blocked requests continue | Spoofed agents, cached rules, or Amzn-User behavior | Validate the address and agent before drawing a conclusion |
| Google pages lose snippets | An unrelated indexing directive changed | Restore the prior rule and audit page-level robots controls |
Limitations of This Configuration
Allowing a crawler creates permission, not a ranking signal or inclusion guarantee. Blocking one agent does not erase material already obtained, control other companies, or secure public files. Robots rules also cannot measure mentions, citations, or assistant exposure. Use logs only to measure identifiable requests, and use analytics only for referred visits that actually reach the site.
Review the policy when Amazon updates its documentation, when your content rights change, or when a plugin replaces robots.txt output. Keep a dated copy of each version so future troubleshooting has a reliable baseline.
Conclusion
A reliable Amazonbot WordPress setup begins with classification, not a blanket Allow or Disallow. Separate Amazonbot, Amzn-SearchBot, and Amzn-User by their documented purposes, publish explicit rules through the layer that controls the live file, and verify claimed traffic against Amazon’s IP lists. That produces an auditable policy without promising search inclusion or AI citations.
Frequently Asked Questions
Does blocking Amazonbot also block Amzn-SearchBot?
No. Amazon documents the three user-agent settings as independent, so each agent should receive its own rule when you need a clear policy.
Does Amzn-SearchBot train generative AI models?
Amazon states that Amzn-SearchBot does not crawl content for generative AI model training. Its documented purpose is improving search experiences in Amazon products and services.
Will allowing Amzn-SearchBot make my pages appear in Alexa?
No. Allowing access can make content eligible for supported search experiences, but it does not guarantee crawling, indexing, selection, citation, or traffic.
Can robots.txt fully block Amzn-User?
No reliable guarantee exists. Amazon says Amzn-User may not follow all robots.txt directives because its requests can be initiated by a user.
How do I know whether an Amazon crawler request is genuine?
Match the recorded source address to the official IP list for the claimed agent. Do not trust the user-agent string by itself because it can be copied.