CCBot WordPress setup is a robots.txt decision: allow Common Crawl, block it sitewide, or limit selected paths. This guide shows how to choose a policy, preserve Googlebot access, and verify whether claimed CCBot requests are authentic.
Quick answer
To block CCBot across a WordPress site, add a dedicated robots.txt group containing User-agent: CCBot and Disallow: /. To allow it, remove that block or use an empty Disallow: line. Always check the public robots.txt response after editing. Common Crawl states that CCBot checks robots.txt before fetching pages and publishes current IP ranges for verification.
Key takeaways
- CCBot is Common Crawl’s crawler, not an AI assistant, search ranking signal, or referral source.
- A CCBot rule can be changed without blocking Googlebot when it is placed in its own user-agent group.
- robots.txt manages cooperative crawler access; it is not access control for private material.
- A matching user-agent string alone does not prove a request came from Common Crawl.
- Blocking future crawling does not remove copies already present in an archive or downstream dataset.
What CCBot does
Common Crawl describes CCBot as the crawler used to build its openly accessible web crawl repository. Its current user agent begins with CCBot/2.0. The organization’s official FAQ says the crawler checks robots.txt, then sends an HTTP GET when access is allowed. It does not currently execute JavaScript or use cookies.
| Item | What it means | What it does not prove |
|---|---|---|
| CCBot request | A crawler requested a URL | That a specific AI system used or cited the page |
| Common Crawl archive | A public dataset may contain a captured page | That every downstream dataset contains it |
| AI assistant referral | A visitor arrived through a trackable link | That all mentions or citations are measured |
Server logs measure crawl requests, while analytics may measure identifiable visits. Neither records all AI mentions, citations, or downstream data use.
Prerequisites for a safe CCBot WordPress setup
- Administrator access to the SEO plugin, hosting panel, or code location that controls robots.txt.
- Permission to change crawler policy for the domain.
- Access to
https://yourdomain.com/robots.txtin a private browser window. - Server or CDN logs if you plan to authenticate CCBot requests.
- A backup of the current rules before making changes.
Read the complete live file first. WordPress, an SEO plugin, or the server may produce it. Edit the layer that serves the public response. The WordPress robots_txt filter affects WordPress-generated output, not a physical file served earlier.
Step 1: Choose an access policy
| Policy | Use it when | Rule |
|---|---|---|
| Allow | You accept eligible public pages being considered for Common Crawl | Remove a CCBot block or use an empty Disallow |
| Block all | You do not want future CCBot page fetches | Disallow the root path |
| Limit paths | You want public articles crawlable but not low-value sections | Disallow only named paths |
Choose the access you actually want. Common Crawl data is open for research and analysis, so allowing CCBot may broaden archival availability; blocking it limits future compliant fetches.
Step 2: Add the correct robots.txt rule
Block CCBot sitewide
User-agent: CCBot
Disallow: /This is the exact blocking pattern published by Common Crawl. Keep it in its own group so rules meant for CCBot do not accidentally apply to every crawler.
Allow CCBot explicitly
User-agent: CCBot
Disallow:An explicit allow group is optional when no broader rule blocks CCBot, but it can make the policy easy to audit.
Block selected WordPress paths
User-agent: CCBot
Disallow: /wp-admin/
Disallow: /search/
Disallow: /tag/Path rules are prefix-based instructions, so test them against real URLs. Do not block CSS, JavaScript, images, or important article paths without a clear reason. For broader syntax guidance, use this robots.txt guide.
Step 3: Verify the public result
- Open the root robots.txt URL, not a WordPress editor preview.
- Search for every
User-agent: CCBotgroup and check for conflicting duplicates. - Confirm sitemap lines and unrelated crawler rules still appear.
- Purge the relevant page or CDN cache if the old response remains.
- Test from a logged-out browser and, when possible, with a command-line request.
curl -i https://example.com/robots.txtLook for HTTP 200 and plain-text rules on the canonical host. A direct response is easiest to audit. Compare generated-file behavior with the WordPress AI search snippet controls guide; robots access and snippet controls solve different problems.
Step 4: Authenticate CCBot in server logs
User-agent strings can be copied. Common Crawl warns about crawlers falsely identifying themselves as CCBot and publishes current IPv4 and IPv6 ranges in a machine-readable CCBot IP JSON file. Treat that file as the current source rather than hard-coding IP ranges from an old article.
- Record the source IP for the log request.
- Compare it with the current published CIDR ranges.
- For IPv4, run a reverse DNS lookup and check that the hostname is under
crawl.commoncrawl.org. - Resolve that hostname forward and confirm it returns the original IP.
- For IPv6, rely on the published range because Common Crawl says reverse DNS is not yet supported there.
This forward-confirmed reverse-DNS pattern is also useful when verifying other crawlers. The practical workflow in the OAI-SearchBot log verification guide shows how to avoid trusting a user agent by itself.
Worked example: a publisher allows articles but excludes utility pages
Imagine a tutorial site wants lessons available to Common Crawl but excludes search results and tag archives. The administrator confirms that an SEO plugin generates robots.txt, saves the current output, and blocks /search/ and /tag/ only for CCBot. The live file still shows unchanged Googlebot rules.
When a later log entry claims to be CCBot, the team checks its IP against the live JSON file and completes the DNS verification. It records the authenticated crawl separately from analytics and does not treat it as an AI citation.
Troubleshooting checklist
| Problem | Likely cause | Next check |
|---|---|---|
| Rule is missing publicly | Wrong editor or physical file override | Identify which layer serves robots.txt |
| Old rule still appears | Plugin, host, or CDN cache | Purge only the relevant cache and retest |
| CCBot still checks robots.txt | Normal policy refresh | Common Crawl says it periodically rechecks the file |
| Unexpected pages are fetched | Path mismatch, duplicate group, or spoofed bot | Audit syntax and authenticate the source IP |
| Google crawling changes | Rule placed under the wrong user-agent group | Restore the backup and separate crawler groups |
If you manage several AI-related crawlers, document each independently. Crawler names that belong to one company can still have different purposes, as shown in the Meta crawler comparison and the Amazon crawler setup guide.
Limitations and measurement
robots.txt is a cooperative crawling instruction, not a security boundary. Google’s robots.txt documentation explains that rules cannot enforce crawler behavior and should not protect confidential content. Use authentication or server access controls for private material.
A new block governs future compliant requests; it does not reliably erase historical captures or copies already obtained by others. Common Crawl also offers an opt-out registry, but that is a separate process from editing WordPress. Finally, do not use a CCBot log line as a GEO success metric. Track authenticated crawl activity, identifiable assistant referrals, search performance, and any reproducible citation checks as separate measurements.
Frequently Asked Questions
Does blocking CCBot block Googlebot?
No. A rule in a dedicated CCBot user-agent group applies to CCBot, not Googlebot. Check the complete live file to ensure the groups were not accidentally combined.
Is CCBot the same as an AI assistant?
No. CCBot is the crawler used by Common Crawl to build an open web archive. Its visit does not prove that an assistant mentioned, cited, or used a page.
Can robots.txt remove pages already stored by Common Crawl?
No. A robots.txt change controls future compliant crawling. It does not guarantee deletion from earlier archives or from datasets that another party already obtained.
Why does CCBot still request robots.txt after I block it?
Common Crawl says it periodically checks robots.txt for policy changes. A request for the policy file is therefore not proof that blocked article pages are still being fetched.
How can I tell whether a CCBot request is genuine?
Compare the source IP with Common Crawl’s current published ranges. For IPv4, also perform reverse and forward DNS checks and confirm that both results return to the original address.
Conclusion
A reliable CCBot WordPress setup starts with a deliberate access policy, a crawler-specific robots.txt group, and verification of the public response. If you inspect logs, authenticate the source rather than trusting its label. This workflow gives you practical control over future Common Crawl access without confusing crawler activity with Google indexing, AI citations, or measurable visitors.