Blog · 11 min read
Is your website visible to AI? A guide for business owners
Can ChatGPT, Gemini and Claude see your website? Find out which AI assistants can access your site, how AI crawlers work, and how to control them.

Summarise this article with
Key takeaways
- AI companies run different bots for different jobs: training their models, building their search, and fetching a page when someone asks.
- OpenAI and Anthropic run search and training as separate bots, so you can allow one and block the other. Perplexity says its search bot is not used for training.
- If you want customers to find you, allow the search bots and make a separate choice about the training bots.
- robots.txt is a request, not a lock. Bots that fetch pages for a user may skip it, and so do unidentified scrapers.
- Check your firewall or CDN bot settings too: they can block AI crawlers without a line in robots.txt.
Can ChatGPT, Gemini, Claude and Grok see your website?
Usually yes, unless your robots.txt file or your hosting blocks them. Most AI assistants read the web through named bots, and those names are what your robots.txt allows or blocks. This is where each one gets its view of your site.
| AI assistant | What reads your site | Can you control it in robots.txt? |
|---|---|---|
| ChatGPT | OAI-SearchBot, GPTBot and ChatGPT-User |
Yes, by name, with one exception (see the list below) |
| Gemini | Googlebot for Google Search; Google-Extended is the robots.txt name for Gemini training and grounding |
Mostly: Google says fetches a user asks for, such as Gemini Notebook, generally ignore robots.txt |
| Claude | Claude-SearchBot, ClaudeBot and Claude-User |
Yes |
| Perplexity | PerplexityBot and Perplexity-User |
Mostly (see the list below) |
| Microsoft Copilot | Bingbot |
Yes, but blocking it also removes you from Bing |
| Grok | No published bot names | No: we found no crawler page from xAI in November 2026 |
The rest of this guide explains what each bot does and which ones a business that wants customers should let in.
What do AI crawlers do on your website?
AI crawlers are programs that AI companies send to read web pages. They do one of three jobs: collect pages to train a model, build the index an AI search tool answers from, or fetch a page because someone asked the assistant a question. Each job usually has its own bot, with its own name.
That split matters because you can treat each job differently. A bot that builds an AI search index can send people to your site with a link. A training bot collects pages that may be used to train a model, which sends you no visitors. A user-triggered bot reads one page for one person, at that moment.
Each named bot announces itself with a name, called a user agent. Your robots.txt file, at the root of your site (for example example.com/robots.txt), uses those names to tell each bot what it may read.
Training
Collects pages that may be used to train a model. Sends you no visitors.
GPTBotClaudeBotGoogle-Extendedrobots.txt: your choice
Search
Builds the index an AI assistant answers from, and links back to your pages.
OAI-SearchBotClaude-SearchBotPerplexityBotBingbotrobots.txt: allow to be found
Fetching for a user
Reads one page when someone asks the assistant about it.
ChatGPT-UserClaude-UserPerplexity-Userrobots.txt: some skip it
AI crawler list: who they are and what each one does
The main AI companies each publish a list of their bots. These are the names that matter for most business sites, in each company’s own description.
| Company | Bot | Its job | Can robots.txt control it? |
|---|---|---|---|
| OpenAI | OAI-SearchBot |
Finds pages to show in ChatGPT search answers | Yes |
| OpenAI | GPTBot |
Collects pages that may be used to train OpenAI’s models | Yes |
| OpenAI | ChatGPT-User |
Visits a page when a ChatGPT user asks it to | May not apply, says OpenAI |
| Anthropic | Claude-SearchBot |
Improves the quality of Claude’s search results | Yes |
| Anthropic | ClaudeBot |
Collects pages that may be used to train Claude | Yes |
| Anthropic | Claude-User |
Fetches a page when a Claude user asks a question | Yes, says Anthropic |
| Perplexity | PerplexityBot |
Finds and links pages in Perplexity’s search results; not used for training | Yes |
| Perplexity | Perplexity-User |
Visits a page to answer a user’s question | Generally no, says Perplexity |
Google-Extended |
Not a separate bot: a robots.txt name that controls whether Gemini may use your pages for training and grounding | Yes: it is only a robots.txt name | |
| Microsoft | Bingbot |
Crawls for Bing search; Microsoft’s AI answers, including Copilot, draw on what it crawls | Yes |
Two of these are easy to miss. Google-Extended has no effect on Google Search: Google says it is not used for inclusion or ranking there. And there is no separate Microsoft AI bot to block. Bingbot does both jobs, so blocking it takes you out of Bing search as well.
Other companies collect pages that may be used for AI training too. Amazon says Amazonbot may be used to train its AI models. Meta says meta-externalagent is for training its AI models or indexing content. Common Crawl’s CCBot builds an open archive of the web for research. Apple works like Google: Applebot-Extended is a robots.txt name for opting out of training, while Applebot keeps crawling for Siri, Spotlight and Safari. To block any of these, add its name to the training group in the example below.
Which AI crawlers should you allow or block?
For most businesses that want new customers, allow the search bots, because they are how AI assistants find your pages and link to them. Then make a separate, deliberate choice about the training bots. Blocking those asks the companies to leave your pages out of future training, and OpenAI says it has no effect on ChatGPT search.
OpenAI states this directly: each setting is independent, so a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Anthropic splits its bots the same way. Perplexity has no training bot: it says PerplexityBot is not used for training.
Some businesses do have a reason to block more. If you sell content people pay to read, or your income depends on page views from your own articles, you may decide the trade is not worth it; robots.txt only records the decision.
Avoid blocking by accident. The usual causes are a broad rule meant to stop scrapers, a Disallow: / left over from a staging site, and a ready-made “block all AI bots” list. One open-source list on GitHub, for example, also names OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot and Applebot, so pasting it in takes you out of AI search answers as well. Check every name against the table above before you add it.
Broad rules are common. Zyte checked the landing pages of 11,100 popular sites in 2026 and found 39.9% blocked AI crawlers that follow robots.txt. Only 12.7% named an AI bot; the other 27.2% blocked every bot with a single User-agent: * rule.
How do you allow or block AI crawlers in robots.txt?
You add a group for each set of bots: one or more User-agent lines naming the bots, then Allow or Disallow lines saying which paths they may read. Disallow: / blocks the whole site. A bot that finds a group with its own name follows that group and ignores the general * group.
Example: allow AI search, block AI training
This example allows the AI search bots, blocks the training bots and leaves everything else open:
# AI search: allowed, so assistants can find and link to us
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
# AI training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
Allowed
AI search bots can find your pages and link to them.
OAI-SearchBot Claude-SearchBot PerplexityBot
Blocked
AI training bots are asked to leave your pages out.
GPTBot ClaudeBot Google-Extended
Unchanged
Every other bot follows the * group, so search engines crawl as before.
Googlebot Bingbot
To allow everything instead, you need no AI lines at all: a bot that finds no group with its name follows the * group. To ask every crawler to skip an area such as /admin/, add Disallow: /admin/ to each group, because a bot that finds a group with its name ignores the * group.
How long changes take
Changes are not instant. OpenAI says its search results can take about 24 hours to reflect a robots.txt change, and Perplexity says up to 24 hours.
Do AI bots respect robots.txt?
robots.txt is a public request, not a lock. Well-behaved crawlers read it and follow it. Bots that fetch a page because a person asked may not: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Unidentified scrapers can ignore it completely.
So use robots.txt to tell the named AI companies what you want, not to keep anything private. Anything that must stay private belongs behind a login.
Search engines and AI companies are also adding finer controls. Microsoft, for example, will honour separate preferences for AI training and for AI grounding (using your page to support an answer) from 1 February 2027. Until a site sets them, both default to Allow, and Microsoft says limiting them does not remove a site from Bing search results. A shared robots.txt name, __usage_ai-training, is also arriving: a Disallow line under it asks for no AI training. Anthropic began recognising it on 30 October 2026, Bing will from 1 February 2027 and Google plans to in early 2027. Anthropic says a group that names ClaudeBot still takes priority.
What else can stop AI crawlers reading your site?
Two things block AI crawlers without anything in robots.txt.
robots.txt
Asks named bots to stay out of all or part of your site.
Firewall or CDN
Can turn a bot away before it reads anything, whatever robots.txt says.
The page itself
If the text appears only after JavaScript runs, some crawlers see an almost empty page.
Page read
The crawler gets your text and can use it in answers.
Bot settings in your firewall or CDN
Some hosting, CDN and security services can block AI bots with a switch in their dashboard. That block happens before the bot ever sees your robots.txt, so the file can say “allow” while the bot is turned away. If you or your developer turned on a bot-blocking feature, check what it covers.
Cloudflare is a common example. Its dashboard has separate controls for AI search, AI agents and AI training. Since 15 September 2026, setting Training to Block also blocks Googlebot, Bingbot and Applebot, because they collect pages for both search and training, so the site drops out of those search engines too. To refuse training and stay in search, Cloudflare’s option is Disallow AI Training. Its older Block AI Bots switch has been replaced by these controls.
Text that only appears after JavaScript runs
Not every crawler runs JavaScript. If your page builds its text in the browser, a crawler that reads only the HTML your server sends may see an almost empty page. Sending the main text in the HTML itself is the safe choice for search engines and AI crawlers alike. It is worth asking about whenever you build or rebuild a website.
How do you check which AI crawlers your site blocks?
Start with your robots.txt: open yourdomain.com/robots.txt in a browser and read each group against the table above. Look for a User-agent: * group with Disallow: /, which blocks every bot that has no group of its own.
Then confirm it from outside. Our free website audit reads your robots.txt and reports which of GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and Bingbot it blocks. It also checks whether your page text is in the HTML. Your hosting’s access logs show which bots actually visited, by user agent name. Google Analytics will not show them: Google says traffic from known bots is excluded automatically.
A user agent name is only what the visitor claims, and anyone can copy it. To confirm a visit really came from a company, check its IP address against the ranges the company publishes, or run the reverse DNS check it describes. Google, Apple and Common Crawl explain how.
How often should you review these rules?
Review your robots.txt about twice a year, and whenever you change hosting, CDN or security settings. AI companies add and rename bots, and the rules change: Microsoft announced its new AI preferences on 30 October 2026. Each company’s own crawler page is the place to check the current names.
Being readable is the first step, not the result: an assistant that can read your pages still decides for itself whether to mention or link to you.
If you would rather have this set up and checked for you, our AI visibility service covers crawler access, indexing in Bing and Google, structured data, and tests of real customer questions in AI assistants.
About this content
Websites, SEO and AI visibility
Written by the Stackwillow editorial team. Every outside fact comes from the company’s own documentation or a named study. We update posts when the rules they describe change.