ClaudeBot, Claude-SearchBot, and Claude-User serve different purposes and should be controlled separately in WordPress. ClaudeBot collects public web content that may contribute to future model training. Claude-SearchBot supports search-result quality. Claude-User retrieves content when a Claude user asks for it. Anthropic says all three honor robots.txt, so publishers can allow search access while declining training collection.
Quick Answer
Choose a policy for each Anthropic bot instead of applying one broad rule. A publisher focused on AI-search discovery might block ClaudeBot, allow Claude-SearchBot, and allow Claude-User. Place the three specific groups in the live root robots.txt file, preserve Googlebot and sitemap rules, verify the public response, and check server logs. Do not use IP blocking as the opt-out method because Anthropic says it may prevent its bots from reading your robots.txt preferences.
Key Takeaways
- ClaudeBot controls potential collection for future Anthropic model-training datasets.
- Claude-SearchBot navigates the web to improve the relevance and accuracy of search responses.
- Claude-User fetches pages in response to user-initiated requests.
- Anthropic documents all three bots as honoring standard robots.txt directives.
- Restricting ClaudeBot applies to future materials and is not described as deleting earlier training data.
- Allowing search or user access does not guarantee citations, mentions, referrals, or rankings.
ClaudeBot vs Claude-SearchBot vs Claude-User
| Bot | Primary use | Effect of blocking |
|---|---|---|
| ClaudeBot | Collect content that could contribute to model training | Signals that future site materials should be excluded from training datasets |
| Claude-SearchBot | Analyze web content to improve Claude search responses | Prevents indexing for Anthropic search optimization |
| Claude-User | Retrieve a page for a user’s current question | Prevents user-directed retrieval from the site |
Anthropic explains these roles in its current web-crawler documentation. The distinction matters: blocking training collection does not require blocking search discovery or user-requested access.
Prerequisites Before Editing WordPress
- Access to the public robots.txt configuration for the exact hostname.
- A backup of the current file, including every Sitemap line.
- A written allow or block decision for each Anthropic bot.
- Read-only access to web-server, CDN, or firewall logs.
- One public article and one intentionally restricted path for verification.
WordPress may serve a virtual robots.txt file until a physical file, plugin, host tool, or deployment process overrides it. Determine which layer owns the live response before editing anything. The result at the root URL is authoritative, not the text shown inside a dashboard field.
Step 1: Choose a Three-Part Policy
Start with content governance rather than a copied snippet. Decide whether Anthropic may use public pages for potential training, search discovery, and retrieval requested by a Claude user. A common policy for a public educational site is to decline training collection while preserving search and user-directed access:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /This is an illustrative policy, not a universal recommendation. A licensing agreement, member-only product, regulatory obligation, or private documentation area may require different access. Record who approved the decision and why.
Step 2: Merge Rules Without Blocking Search
Open the current file and add only the necessary Anthropic groups. Do not replace existing Googlebot, OAI-SearchBot, PerplexityBot, administrative-path, or Sitemap rules. The robots.txt guide explains root placement and group structure.
Google’s robots.txt documentation notes that a file applies only to the protocol, host, and port where it is published. If the site uses separate subdomains, audit each one rather than assuming the main-domain rule covers them.
Use explicit groups because a broad wildcard rule may affect other crawlers. Keep capitalization consistent with Anthropic’s documented tokens and avoid duplicate groups that make future reviews harder.
Step 3: Use Crawl-Delay Carefully
Anthropic says it supports the non-standard Crawl-delay extension when publishers need to limit activity. For example:
User-agent: ClaudeBot
Crawl-delay: 1This directive is not universal. Google does not support Crawl-delay in robots.txt, so do not treat it as a sitewide performance control. Apply it only to the intended Anthropic group, monitor actual request patterns, and address origin-capacity problems separately.
If crawling causes server strain, first confirm the requesting identity and response volume. A temporary traffic spike, a spoofed user-agent, or an unrelated bot should not trigger a permanent policy change for a legitimate crawler.
Step 4: Verify the Live File and Logs
- Open the public root robots.txt URL in a private browser window.
- Confirm a successful plain-text response from the correct hostname.
- Check that each Anthropic token has the intended allow or disallow rule.
- Verify that Googlebot, sitemap, and other established rules remain unchanged.
- Search logs for ClaudeBot, Claude-SearchBot, and Claude-User requests.
- Record timestamps, URLs, status codes, source addresses, and firewall actions.
Anthropic publishes a current bot IP list for identity checking. Use it as corroborating evidence with the user-agent and request pattern. Anthropic specifically warns that IP blocking may not reliably preserve an opt-out because blocked bots cannot read the robots.txt file.
Illustrative Worked Example
A WordPress learning site wants its tutorials available to Claude search and user-requested retrieval, but its editorial policy excludes future training collection. The administrator adds a ClaudeBot sitewide disallow plus explicit allows for Claude-SearchBot and Claude-User. Existing Google and sitemap lines are preserved.
| Verification | Expected result | Priority |
|---|---|---|
| ClaudeBot on a public article | Disallowed by robots.txt | High |
| Claude-SearchBot on a public article | Allowed | High |
| Claude-User on a public article | Allowed | High |
| Member-only page | Protected by authentication | Critical |
| Googlebot and sitemap rules | Unchanged | Critical |
The private member area remains protected by authentication, not robots.txt. The administrator then reviews logs after crawlers have had time to read the updated file. This demonstrates policy verification; it does not claim the tutorials will be selected in Claude responses.
Troubleshooting Common Problems
The WordPress change is not visible
Clear relevant caches and check whether a physical file, host setting, security plugin, or CDN response overrides the WordPress-generated file.
All Claude bots were blocked accidentally
Restore the intended separate groups from the backup, verify the public response, and document which access categories should remain available.
A claimed Anthropic bot ignores the rule
Compare its address with Anthropic’s current published list and inspect the exact user-agent. A copied header alone does not establish identity.
ClaudeBot was blocked but old content may remain in a model
Anthropic describes the rule as a signal for future materials. Do not promise retroactive removal based only on a robots.txt change.
The site is allowed but receives no Claude referrals
Crawl permission is only one prerequisite. Retrieval, relevance, response generation, user demand, and link selection still determine visible outcomes.
Limitations and Maintenance
Robots.txt is public crawler guidance, not privacy protection or authentication. It cannot secure confidential files, prove that a model used a page, measure citations, or guarantee traffic. Server logs show requests, while analytics show only visits that arrive with identifiable attribution; neither is a comprehensive Claude visibility report.
Recheck the live rules after hosting migrations, security changes, CDN updates, or SEO-plugin replacements. Review Anthropic’s documentation before using old snippets because bot names and policies can change. For other providers, use their own controls: the OpenAI crawler guide, Google crawler guide, and Perplexity crawler guide describe different identities and scopes.
Frequently Asked Questions
What is the difference between ClaudeBot and Claude-SearchBot?
ClaudeBot collects public content that may contribute to future model training, while Claude-SearchBot supports the quality of Claude search responses.
What does Claude-User do?
Claude-User retrieves web content when a person asks Claude a question that requires access to a page.
Do Anthropic bots respect robots.txt?
Yes. Anthropic states that ClaudeBot, Claude-SearchBot, and Claude-User honor standard robots.txt directives.
Should I block Anthropic bots by IP address?
Not as the primary opt-out. Anthropic warns that IP blocking may prevent its bots from reading the robots.txt preferences and may not remain reliable.
Will allowing Claude-SearchBot guarantee a citation?
No. Allowing access does not guarantee a citation, mention, ranking, referral visit, or conversion.
Conclusion
A safe Claude crawler setup separates training, search, and user-directed retrieval. Configure ClaudeBot, Claude-SearchBot, and Claude-User according to a documented policy, preserve unrelated WordPress rules, verify the public file, and confirm genuine requests through logs. That approach provides clear governance without treating crawler access as a guaranteed AI-search tactic.