Short answer: Make sure your robots.txt allows OAI-SearchBot, OpenAI's search crawler, and that your host or firewall doesn't block it. Sites that block it can't be shown in ChatGPT search answers. Changes take about a day to reach ChatGPT. The free checker shows what your robots.txt says to each OpenAI bot.

OpenAI's three bots

OpenAI uses three different bots, and each robots.txt setting works on its own (OpenAI crawlers):

  • OAI-SearchBot visits pages so they can appear in ChatGPT search. This is the one that matters: block it and your pages can't be shown in ChatGPT search answers.
  • GPTBot collects pages to train OpenAI's models. Blocking it doesn't take you out of ChatGPT search.
  • ChatGPT-User opens a page when someone in ChatGPT asks it to. It doesn't decide what shows up in search, and OpenAI says robots.txt may not apply to it, because a person asked for the page.

So you can say no to training and still be found: block GPTBot and allow OAI-SearchBot.

Check your robots.txt

Open yoursite.com/robots.txt in your browser. Look for any group that starts with User-agent: OAI-SearchBot, User-agent: *, or a list of AI bots, followed by Disallow: /. Any of those can keep ChatGPT out.

A bot follows only the group that names it, and falls back on User-agent: * if no group does. So a site that blocks every bot with * also blocks OAI-SearchBot, unless OAI-SearchBot has its own group.

The robots.txt that lets ChatGPT search in

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Leave out the GPTBot group if you're happy for your pages to be used for training. Keep the rest of your file as it is.

OpenAI says it can take about 24 hours after you change robots.txt for ChatGPT search to follow the new rules (OpenAI crawlers).

Check your host and firewall too

robots.txt is a request, not a lock. Some hosts, firewalls and services like Cloudflare can block AI bots with a setting, before robots.txt even comes into play. If your robots.txt allows OAI-SearchBot but ChatGPT still never cites you:

  • Look in your host's or CDN's security settings for options about AI bots, crawlers or “bot fight” modes, and allow OAI-SearchBot.
  • If your firewall allows bots by address, OpenAI publishes the addresses OAI-SearchBot uses at openai.com/searchbot.json.

Make sure there is something to read

OpenAI's crawlers download a page's code but don't run its JavaScript (Vercel). If your text only appears after scripts have run in the browser, ChatGPT may see an empty page. A quick test: in your browser, choose “View page source” on one of your pages. If you can read your text there, so can the crawler.

Sources