Many sites added a GPTBot block to their robots.txt file because they do not want their content used for AI training. Then came this question: does this setting also stop ChatGPT from mentioning my brand? The short answer: GPTBot on its own is not about visibility in search answers. But most sites do not block only GPTBot. They block all AI bots at once. That is usually where the real problem comes from.
What are OpenAI's bots?
In its crawler documentation, OpenAI defines three separate user agents. Each does a different job:
- GPTBot: crawls content that may be used to train OpenAI's foundation models. According to the documentation, blocking GPTBot signals that your site's content should not be used to train these models.
- OAI-SearchBot: used to show sites in ChatGPT's search features. According to the documentation, sites that block this bot are not shown in ChatGPT search answers. They may still appear as navigation links only.
- ChatGPT-User: may visit a page when a user asks a question or starts an action in ChatGPT. Because the user starts the action, robots.txt rules may not apply to this bot.
OpenAI states clearly that these settings are independent of each other. For example, you can allow OAI-SearchBot to appear in search results and block GPTBot.
What happens if I block GPTBot?
According to the documentation, a GPTBot block is a request that your content not be used in future model training. Whether you are shown in ChatGPT search answers depends on the OAI-SearchBot setting.
One detail is useful to know here. ChatGPT can answer a question in two ways: from what it learned in training, or from pages it searches and reads on the web at that moment. The GPTBot block relates to the first way, the OAI-SearchBot block to the second. OpenAI does not give a number or explanation about how a GPTBot block affects how well the model knows a brand. So it would not be right to say anything definite about this. You can read Where does ChatGPT get its information?, where we explain the two ways in more detail.
Which setting really needs attention?
On sites whose brand does not appear in ChatGPT searches, a common situation is this: while trying to block GPTBot, OAI-SearchBot was also blocked. This can happen in a few ways:
- robots.txt has a general block for all bots, and only Googlebot is allowed.
- A security plugin has shut them all off at once with a "block AI bots" option.
- The hosting company or CDN stops bot traffic with its own setting.
In the last two cases your robots.txt file may look clean, but the bots still cannot reach your pages. So you need to check not only the file but also your server and security settings.
What does a setting closed to training but open to search look like?
If you do not want your content used for training but want to appear in ChatGPT searches, you can set up this logic in robots.txt:
- For GPTBot: Disallow: /
- For OAI-SearchBot: Allow: /
Do not rush after the change. OpenAI says it can take about 24 hours for a robots.txt update to be reflected in its systems.
This is a matter of choice. Allowing GPTBot and blocking it are both legitimate decisions. What matters is that your decision does not cut your search visibility without you noticing.
What questions should you ask when deciding?
Whether to block GPTBot is a business decision. These questions can guide you:
- Is your content a product you sell? If you publish news, research or paid content, you may want to stay closed to training use.
- Is your site mostly promotional? If the purpose of your service and product pages is to be known anyway, staying open to training may not be a loss for you.
- How important is search visibility to you? If it is important, allow OAI-SearchBot whatever your training decision is.
- Who manages the setting? If you manage robots.txt and someone else manages the security settings, make sure both sides apply the same decision.
What are the common mistakes?
Common mistakes here include:
- Shutting off all AI bots with one line and blocking the search bots too without noticing.
- Fixing robots.txt but forgetting the bot block in the CDN or security plugin.
- Expecting results right after changing the setting. OpenAI and Perplexity say it can take about a day for the change to be reflected.
- Misspelling the bot name. The name in robots.txt must match the user agent name in the documentation exactly.
How do other engines handle this?
A similar split exists in other engines:
- Perplexity: according to Perplexity's documentation, PerplexityBot is used to show and link to sites in Perplexity search results. It does not crawl content for foundation model training. They recommend allowing this bot to appear in search results. Perplexity-User, which visits for user actions, generally ignores robots.txt rules.
- Google: according to Google, Google-Extended is a setting for managing whether crawled content is used to train future Gemini models. Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal.
So in all three engines, "training" and "being shown in search" are separate settings. When you turn one off, you need to check whether you turned off the other too.
How do you check your own site?
For a manual check, you can follow this order:
- Add /robots.txt to the end of your site's address and open the file.
- Search for the names GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended. Note what each one is allowed.
- Check whether there is a general block under "User-agent: *".
- Check whether your security plugin and your CDN panel have a bot blocking setting.
The technical check on AIShortlist's Site screen scans your robots.txt file and whether AI bots are blocked. It looks at common bots, including GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended. It also says what to do for each finding.
Will I become visible once the bot setting is fixed?
Bot access is a prerequisite, not enough on its own. If bots can enter your site but the engines still do not mention you, the cause is elsewhere: for example, your pages do not clearly explain what you do, or your name does not appear on the list pages the engines read. Why doesn't ChatGPT recommend my brand?, where we go through these causes one by one, is a good place to start for the next step.
After you fix the setting, you need to measure to see its effect. With the free trial you can measure 10 questions once in ChatGPT, Gemini and Perplexity, and see the site check in the same account. No card is required.