THE SHORT ANSWER
Anthropic does not operate a single crawler. ClaudeBot is the broad web crawler associated with model training, Claude-User is the user-initiated fetch that happens when someone in a chat asks about a specific page, and Claude-SearchBot supports search-style retrieval. They carry separate user-agent tokens, so a robots.txt rule aimed at one leaves the others untouched, and blocking the training crawler does not remove you from live answers.
Most robots.txt files treat Anthropic as one thing. That was roughly true in 2023 and has not been true for a while. There are several agents behind the name, they do different jobs, and the token you block determines whether you disappear from a training corpus, from a live citation, or from nothing at all because you blocked an agent that was never coming anyway.
Worth stating up front: crawler naming at every AI vendor has churned more than the published documentation suggests, and Anthropic is no exception here. Older tokens still circulate in guides that were perfectly correct on the day they were written and now describe an agent that no longer behaves that way. Read the current vendor documentation before copying a rule out of a blog post.
The numbers, at a glance
ClaudeBot: the broad crawler associated with corpus building; blocking it affects future model versions, not today's answers
Claude-User: fires when a person in a conversation asks about a specific page, so it behaves more like a browser than a crawler
Claude-SearchBot: supports search-style retrieval, which is the agent that matters if you want to be cited in a live answer
Legacy tokens: anthropic-ai and Claude-Web appear in older guides; treat any rule you inherited as needing verification rather than trust
What each agent does, and which one you actually care about
ClaudeBot. General web crawling on Anthropic's behalf. It declares itself in the user-agent header and honours robots.txt directives written against that token. Its output feeds corpus work, which means the effect of allowing or blocking it lands months later, on a model release schedule rather than a publishing schedule.
Claude-User. Triggered by a human action. Somebody in a conversation pastes a link or asks what a page says, and the fetch goes out for that one URL. Vendors generally classify this class of request as user-initiated rather than automated, which is why the robots.txt treatment of it differs from a bulk crawler.
Claude-SearchBot. Retrieval-facing. If your commercial goal is being named and linked inside an answer somebody reads this week, this is the agent whose access decides it.
The practical consequence: a business that wants to be described accurately should be reachable by the retrieval agent regardless of what it decides about the training crawler. Those are separate questions and deserve separate answers rather than one blanket line.
How to verify a request is genuinely from Anthropic
A user-agent string is self-reported text. Anyone scraping your site can put ClaudeBot in the header, and plenty of low-grade scrapers do exactly that to inherit whatever allowances you granted. Server logs full of ClaudeBot hits therefore prove nothing on their own.
The verification method is the same one you would use for Googlebot. Take the client IP from the log line and check it against the published address ranges the vendor maintains for its crawlers, or do a reverse lookup and then a forward lookup on the resulting hostname to confirm the pair agrees. Requests that fail this are impersonators, and they are the ones worth rate-limiting. Requests that pass are the ones your robots.txt rules were written for.
What blocking actually achieves, and what it does not
Blocking the training crawler prevents future fetches. It does not reach backwards into a model that has already been trained, and it does not erase what other sites say about you. A company that blocks its own domain and leaves its directory listings, review profiles and competitor comparison pages untouched has not become invisible. It has handed the description of itself to third parties.
Blocking the retrieval agent is a different and more immediate decision, because it removes the mechanism by which a current page corrects an out-of-date impression. For a publisher whose archive is the product, that trade can still be rational. For a contractor whose website exists precisely to explain what they do and where they work, it very rarely is.
The rule you probably want, and the checks around it
Name the tokens separately. Write one user-agent group per agent rather than relying on a wildcard, so the decision is legible to whoever reads the file next year.
Keep any disallow path-scoped. Withholding a client-work directory is a small decision. Withholding the root is a large one that people make by accident.
Check the edge. Bot-management categories on CDN platforms drop AI agents before robots.txt is ever parsed, so a permissive file can coexist with a hard block.
Read your own logs. If a verified Anthropic agent has never appeared, the problem is access or discoverability, not content.
Re-read after every migration. Platform moves lose robots.txt more reliably than any other file on the site.
Auditing your Anthropic crawler posture
Grep the last ninety days of access logs for each agent token and count verified hits per day.
Confirm the hits by IP range or reverse-forward DNS before you believe the user-agent string.
Write separate robots.txt groups for the training agent and the retrieval agent.
Open the CDN bot-management panel and confirm the AI category is not silently rejecting them.
Diary a six-month review, because vendor tokens and documentation change without notice.
Want leads like this in your pipeline?
Flock runs the campaigns, screens the enquiries and hands you only the ones that match your service area, job size and capacity. You pay per lead, not per month.
Book a 15-minute fit check | See lead package pricing
Related answers
Frequently asked questions
Does blocking ClaudeBot remove my company from Claude's answers?
No. It stops future crawling of your pages. Anything already learned stays, and anything published about you elsewhere remains available. In practice the description gets less accurate rather than disappearing, because your own site was the only source you controlled.
Why do I see ClaudeBot in my logs but never get cited?
Two common causes. The hits may be impersonators using the token, which log analysis alone cannot distinguish. Or the crawler reaches you fine and your pages are simply not extractable enough to be the best available passage for the questions people ask.
Is Claude-User bound by robots.txt?
Vendors generally treat user-initiated fetches differently from bulk crawling, on the reasoning that a person requested that specific page. The treatment varies by vendor and has been revised more than once, so check the current documentation rather than assuming parity with the training crawler.
Should a small contractor site care about any of this?
Mainly about access, not policy. Confirm nothing at the CDN or in an inherited robots.txt is blocking retrieval agents, then spend your remaining effort on making the pages worth quoting. The crawler question is a five-minute check, not a project.
NEED A CLEARER PLAN?
Let’s turn your next move into momentum.
Talk to us →