Which AI Crawlers a Criminal Defense Website Should Let In
Every AI company that offers a search feature sends software to read websites. Some of those crawlers collect pages that may be used to train future models. Others fetch pages so an assistant can cite them in an answer. A firm's robots.txt file decides which of them may read the site, and many law firms block or allow all of them without knowing the difference.
For a criminal defense practice that wants to be named when someone asks ChatGPT or Claude about a DUI lawyer in their city, that distinction matters. Blocking the wrong crawler can keep a firm out of AI search answers. Allowing everything may conflict with a firm's own view about training data. This guide explains the main crawlers and how to decide.
Training crawlers and search crawlers are separate
OpenAI publishes a list of its crawlers. GPTBot collects content that may be used to train its foundation models. OAI-SearchBot is used to surface sites in ChatGPT search, and OpenAI says sites that opt out of it will not be shown in ChatGPT search answers, though they may still appear as plain navigational links. ChatGPT-User fetches a page when a person asks ChatGPT to look at it, and OpenAI says it does not crawl the web automatically.
Anthropic follows a similar split. ClaudeBot collects web content that may contribute to model training. Claude-SearchBot crawls to improve search result quality. Claude-User retrieves pages when a user asks Claude a question that needs them. Anthropic's help page says disabling the search and user bots may reduce a site's visibility in user-directed search.
The practical upshot is simple. Each of these bots is controlled separately in robots.txt, so a firm can make different choices for training and for search.
Decide what you want before editing robots.txt
Start with two questions. Do you want your pages available for AI model training? Do you want your firm to appear in AI search answers with a link back to your site?
Many defense firms answer the second question with a firm yes. People do ask assistants who handles drug cases in a given county, and a cited answer can send a visitor straight to a charge page. The first question is more personal. Some lawyers are comfortable with training use, and others would rather opt out. Either answer is reasonable, and the search question does not depend on it.
A sample setup for a firm that wants search but not training
A firm that wants to appear in AI search answers while opting out of training could use rules along these lines:
- Disallow GPTBot and ClaudeBot from the whole site.
- Allow OAI-SearchBot and Claude-SearchBot.
- Leave the user-initiated fetchers, ChatGPT-User and Claude-User, unblocked so assistants can read a page a person asks about.
Check each company's current documentation before you publish the file, because crawler names and behavior change. Perplexity and Google publish their own crawler details as well, and each has its own rules.
Check what your site is already doing
Many firms have robots.txt rules they did not write. A website platform, security plugin or hosting firewall may block AI crawlers by default. Some content delivery networks offer a one-click setting that blocks AI bots across the board, and it is easy to switch on without realizing it also blocks search crawlers.
Do three checks this week. Open yourdomain.com/robots.txt and read it. Ask your web host or developer whether a firewall rule blocks known AI user agents. Then look at your server logs or analytics for visits from these crawlers over the past month. If a search crawler never appears, something may be stopping it.
What robots.txt cannot do
Robots.txt is a request that well-behaved crawlers honor. It does not hide content from people, and it does not remove pages that a model has already seen. It also cannot make an assistant recommend you. Being crawled only means your pages are available. Whether your firm gets named depends on how clearly those pages answer the question, how consistent your name and details are across the web and what others say about you.
Firms that care about AI visibility should spend most of their time on clear charge pages, accurate profiles and honest reviews. Robots.txt is a gate. It has to be open, but it does not bring anyone through it.
For more on AI search and legal marketing tools, the ChatGPT and AI search guides for law firms collect related pieces. If you want to see whether these crawlers actually reach your pages, this walkthrough on measuring AI crawler visits and AI referrals shows how to read the logs.
Questions about AI crawlers
If I block GPTBot, will my firm disappear from ChatGPT search?
According to OpenAI's documentation, GPTBot and OAI-SearchBot are controlled separately. Blocking GPTBot covers training use, while appearing in ChatGPT search depends on OAI-SearchBot.
Does allowing AI crawlers create any ethics issue?
Robots.txt settings are a technical choice, but the content those crawlers read is still lawyer advertising. Keep pages accurate and check your state bar's advertising rules for the content itself.
How often should robots.txt be reviewed?
Look at it whenever you change hosts, platforms or security settings, and at least a few times a year, since AI companies add and rename crawlers.
SBD Marketing works only with criminal defense law firms, one firm per city, and helps them with website design, SEO for the charges they defend, visibility in AI search tools such as ChatGPT, Gemini, Claude and Perplexity, and SEO audits.