Blog · 11 min read

Is your website visible to AI? A guide for business owners

Can ChatGPT, Gemini and Claude see your website? Find out which AI assistants can access your site, how AI crawlers work, and how to control them.

A laptop showing a business website. ChatGPT, Gemini, Claude, Perplexity and Copilot badges connect to it with green ticks, a panel lists three kinds of AI bot, and an unidentified bot is stopped by a glass shield

Summarise this article with

Key takeaways

  • AI companies run different bots for different jobs: training their models, building their search, and fetching a page when someone asks.
  • OpenAI and Anthropic run search and training as separate bots, so you can allow one and block the other. Perplexity says its search bot is not used for training.
  • If you want customers to find you, allow the search bots and make a separate choice about the training bots.
  • robots.txt is a request, not a lock. Bots that fetch pages for a user may skip it, and so do unidentified scrapers.
  • Check your firewall or CDN bot settings too: they can block AI crawlers without a line in robots.txt.

Can ChatGPT, Gemini, Claude and Grok see your website?

Usually yes, unless your robots.txt file or your hosting blocks them. Most AI assistants read the web through named bots, and those names are what your robots.txt allows or blocks. This is where each one gets its view of your site.

AI assistant What reads your site Can you control it in robots.txt?
ChatGPT OAI-SearchBot, GPTBot and ChatGPT-User Yes, by name, with one exception (see the list below)
Gemini Googlebot for Google Search; Google-Extended is the robots.txt name for Gemini training and grounding Mostly: Google says fetches a user asks for, such as Gemini Notebook, generally ignore robots.txt
Claude Claude-SearchBot, ClaudeBot and Claude-User Yes
Perplexity PerplexityBot and Perplexity-User Mostly (see the list below)
Microsoft Copilot Bingbot Yes, but blocking it also removes you from Bing
Grok No published bot names No: we found no crawler page from xAI in November 2026

The rest of this guide explains what each bot does and which ones a business that wants customers should let in.

What do AI crawlers do on your website?

AI crawlers are programs that AI companies send to read web pages. They do one of three jobs: collect pages to train a model, build the index an AI search tool answers from, or fetch a page because someone asked the assistant a question. Each job usually has its own bot, with its own name.

That split matters because you can treat each job differently. A bot that builds an AI search index can send people to your site with a link. A training bot collects pages that may be used to train a model, which sends you no visitors. A user-triggered bot reads one page for one person, at that moment.

Each named bot announces itself with a name, called a user agent. Your robots.txt file, at the root of your site (for example example.com/robots.txt), uses those names to tell each bot what it may read.

  • Training

    Collects pages that may be used to train a model. Sends you no visitors.

    GPTBot ClaudeBot Google-Extended

    robots.txt: your choice

  • Search

    Builds the index an AI assistant answers from, and links back to your pages.

    OAI-SearchBot Claude-SearchBot PerplexityBot Bingbot

    robots.txt: allow to be found

  • Fetching for a user

    Reads one page when someone asks the assistant about it.

    ChatGPT-User Claude-User Perplexity-User

    robots.txt: some skip it

Three jobs, three kinds of bot. Google-Extended is a robots.txt name, not a bot. Names from each company’s crawler documentation, read in November 2026.

AI crawler list: who they are and what each one does

The main AI companies each publish a list of their bots. These are the names that matter for most business sites, in each company’s own description.

Company Bot Its job Can robots.txt control it?
OpenAI OAI-SearchBot Finds pages to show in ChatGPT search answers Yes
OpenAI GPTBot Collects pages that may be used to train OpenAI’s models Yes
OpenAI ChatGPT-User Visits a page when a ChatGPT user asks it to May not apply, says OpenAI
Anthropic Claude-SearchBot Improves the quality of Claude’s search results Yes
Anthropic ClaudeBot Collects pages that may be used to train Claude Yes
Anthropic Claude-User Fetches a page when a Claude user asks a question Yes, says Anthropic
Perplexity PerplexityBot Finds and links pages in Perplexity’s search results; not used for training Yes
Perplexity Perplexity-User Visits a page to answer a user’s question Generally no, says Perplexity
Google Google-Extended Not a separate bot: a robots.txt name that controls whether Gemini may use your pages for training and grounding Yes: it is only a robots.txt name
Microsoft Bingbot Crawls for Bing search; Microsoft’s AI answers, including Copilot, draw on what it crawls Yes

Two of these are easy to miss. Google-Extended has no effect on Google Search: Google says it is not used for inclusion or ranking there. And there is no separate Microsoft AI bot to block. Bingbot does both jobs, so blocking it takes you out of Bing search as well.

Other companies collect pages that may be used for AI training too. Amazon says Amazonbot may be used to train its AI models. Meta says meta-externalagent is for training its AI models or indexing content. Common Crawl’s CCBot builds an open archive of the web for research. Apple works like Google: Applebot-Extended is a robots.txt name for opting out of training, while Applebot keeps crawling for Siri, Spotlight and Safari. To block any of these, add its name to the training group in the example below.

Which AI crawlers should you allow or block?

For most businesses that want new customers, allow the search bots, because they are how AI assistants find your pages and link to them. Then make a separate, deliberate choice about the training bots. Blocking those asks the companies to leave your pages out of future training, and OpenAI says it has no effect on ChatGPT search.

OpenAI states this directly: each setting is independent, so a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Anthropic splits its bots the same way. Perplexity has no training bot: it says PerplexityBot is not used for training.

Some businesses do have a reason to block more. If you sell content people pay to read, or your income depends on page views from your own articles, you may decide the trade is not worth it; robots.txt only records the decision.

Avoid blocking by accident. The usual causes are a broad rule meant to stop scrapers, a Disallow: / left over from a staging site, and a ready-made “block all AI bots” list. One open-source list on GitHub, for example, also names OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot and Applebot, so pasting it in takes you out of AI search answers as well. Check every name against the table above before you add it.

Broad rules are common. Zyte checked the landing pages of 11,100 popular sites in 2026 and found 39.9% blocked AI crawlers that follow robots.txt. Only 12.7% named an AI bot; the other 27.2% blocked every bot with a single User-agent: * rule.

How do you allow or block AI crawlers in robots.txt?

You add a group for each set of bots: one or more User-agent lines naming the bots, then Allow or Disallow lines saying which paths they may read. Disallow: / blocks the whole site. A bot that finds a group with its own name follows that group and ignores the general * group.

Example: allow AI search, block AI training

This example allows the AI search bots, blocks the training bots and leaves everything else open:

# AI search: allowed, so assistants can find and link to us
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# AI training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Allowed

AI search bots can find your pages and link to them.

OAI-SearchBot Claude-SearchBot PerplexityBot

Blocked

AI training bots are asked to leave your pages out.

GPTBot ClaudeBot Google-Extended

Unchanged

Every other bot follows the * group, so search engines crawl as before.

Googlebot Bingbot

What the example file asks each bot to do.

To allow everything instead, you need no AI lines at all: a bot that finds no group with its name follows the * group. To ask every crawler to skip an area such as /admin/, add Disallow: /admin/ to each group, because a bot that finds a group with its name ignores the * group.

How long changes take

Changes are not instant. OpenAI says its search results can take about 24 hours to reflect a robots.txt change, and Perplexity says up to 24 hours.

Do AI bots respect robots.txt?

robots.txt is a public request, not a lock. Well-behaved crawlers read it and follow it. Bots that fetch a page because a person asked may not: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Unidentified scrapers can ignore it completely.

So use robots.txt to tell the named AI companies what you want, not to keep anything private. Anything that must stay private belongs behind a login.

Search engines and AI companies are also adding finer controls. Microsoft, for example, will honour separate preferences for AI training and for AI grounding (using your page to support an answer) from 1 February 2027. Until a site sets them, both default to Allow, and Microsoft says limiting them does not remove a site from Bing search results. A shared robots.txt name, __usage_ai-training, is also arriving: a Disallow line under it asks for no AI training. Anthropic began recognising it on 30 October 2026, Bing will from 1 February 2027 and Google plans to in early 2027. Anthropic says a group that names ClaudeBot still takes priority.

What else can stop AI crawlers reading your site?

Two things block AI crawlers without anything in robots.txt.

  1. robots.txt

    Asks named bots to stay out of all or part of your site.

  2. Firewall or CDN

    Can turn a bot away before it reads anything, whatever robots.txt says.

  3. The page itself

    If the text appears only after JavaScript runs, some crawlers see an almost empty page.

  4. Page read

    The crawler gets your text and can use it in answers.

Three places an AI crawler can be stopped before it reads your page.

Bot settings in your firewall or CDN

Some hosting, CDN and security services can block AI bots with a switch in their dashboard. That block happens before the bot ever sees your robots.txt, so the file can say “allow” while the bot is turned away. If you or your developer turned on a bot-blocking feature, check what it covers.

Cloudflare is a common example. Its dashboard has separate controls for AI search, AI agents and AI training. Since 15 September 2026, setting Training to Block also blocks Googlebot, Bingbot and Applebot, because they collect pages for both search and training, so the site drops out of those search engines too. To refuse training and stay in search, Cloudflare’s option is Disallow AI Training. Its older Block AI Bots switch has been replaced by these controls.

Text that only appears after JavaScript runs

Not every crawler runs JavaScript. If your page builds its text in the browser, a crawler that reads only the HTML your server sends may see an almost empty page. Sending the main text in the HTML itself is the safe choice for search engines and AI crawlers alike. It is worth asking about whenever you build or rebuild a website.

How do you check which AI crawlers your site blocks?

Start with your robots.txt: open yourdomain.com/robots.txt in a browser and read each group against the table above. Look for a User-agent: * group with Disallow: /, which blocks every bot that has no group of its own.

Then confirm it from outside. Our free website audit reads your robots.txt and reports which of GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and Bingbot it blocks. It also checks whether your page text is in the HTML. Your hosting’s access logs show which bots actually visited, by user agent name. Google Analytics will not show them: Google says traffic from known bots is excluded automatically.

A user agent name is only what the visitor claims, and anyone can copy it. To confirm a visit really came from a company, check its IP address against the ranges the company publishes, or run the reverse DNS check it describes. Google, Apple and Common Crawl explain how.

How often should you review these rules?

Review your robots.txt about twice a year, and whenever you change hosting, CDN or security settings. AI companies add and rename bots, and the rules change: Microsoft announced its new AI preferences on 30 October 2026. Each company’s own crawler page is the place to check the current names.

Being readable is the first step, not the result: an assistant that can read your pages still decides for itself whether to mention or link to you.

If you would rather have this set up and checked for you, our AI visibility service covers crawler access, indexing in Bing and Google, structured data, and tests of real customer questions in AI assistants.

About this content

Websites, SEO and AI visibility

Written by the Stackwillow editorial team. Every outside fact comes from the company’s own documentation or a named study. We update posts when the rules they describe change.

Questions

Straight answers.

Do I need to do anything for ChatGPT to find my website?

Usually not. If your robots.txt and your hosting’s bot settings do not block OAI-SearchBot, it is allowed to read your pages for ChatGPT search, though OpenAI does not promise any site will appear. Check both once. Our free website audit shows which AI crawlers your robots.txt blocks.

Will blocking GPTBot remove my site from ChatGPT?

No, according to OpenAI. GPTBot collects pages for training, while OAI-SearchBot finds pages for ChatGPT search, and each setting is independent. Blocking OAI-SearchBot is what keeps a site out of ChatGPT search answers, and OpenAI says the site can still appear there as a navigational link.

Does blocking Google-Extended affect my Google ranking?

No. Google says Google-Extended does not affect whether a site is included in Google Search and is not a ranking signal. It only controls whether Gemini may use your pages for training and grounding. Google Search crawls with Googlebot.

Can I block Grok in robots.txt?

Not reliably. In November 2026 we found no crawler page from xAI, the company behind Grok, that names its bots, so there is no documented name to put in robots.txt. If you want to limit bots that publish no name, check the bot settings in your firewall or CDN.

Is robots.txt enough to keep my content away from AI?

No. robots.txt is a public request that named crawlers choose to follow. Bots that fetch a page because a user asked may skip it, and unidentified scrapers can ignore it. Anything that must stay private belongs behind a login.

How long does a robots.txt change take to work?

OpenAI says ChatGPT search can take about 24 hours to reflect a change, and Perplexity says up to 24 hours. Other crawlers re-read robots.txt on their own schedule, so check again after a few days rather than the same hour.

Want this checked on your site? Start with a free quote.

Tell us what you need. We reply within one business day with a plan. Your written quote follows after first contact.

After you write: call notes the same day, then your written quote after our call.