GPTBot

How OpenAI's crawlers divide the work

OpenAI documents three user agents that matter for most websites, each with its own job and its own robots.txt token. Its crawler overview describes them this way:

• GPTBot crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing it "indicates a site's content should not be used in training generative AI foundation models." Example user-agent string (OpenAI notes the version number may change): Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
• OAI-SearchBot surfaces websites in ChatGPT's search features. Sites opted out of it "will not be shown in ChatGPT search answers, though can still appear as navigational links." Its token in the user-agent string is OAI-SearchBot/1.4; +https://openai.com/searchbot.
• ChatGPT-User visits a page when a ChatGPT or custom GPT user asks a question, and handles GPT Actions. OpenAI says it is not used to crawl the web automatically, that "because these actions are initiated by a user, robots.txt rules may not apply," and that it does not determine whether content appears in Search. Full string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

OpenAI states that each setting is independent. A site can allow OAI-SearchBot to appear in search results while disallowing GPTBot. If both are allowed, OpenAI says it may use the results of one crawl for both purposes. For search, a robots.txt change can take about 24 hours to take effect.

OpenAI also publishes IP ranges for each bot, linked from the same page: openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json. When fetching robots.txt files, GPTBot and OAI-SearchBot may add a robots.txt marker to the user-agent string so the requests are easy to spot in logs that omit paths.

Where it came from

OpenAI added GPTBot to its documentation by early August 2023, without a formal announcement. Search Engine Roundtable reported it on Aug 7, 2023, quoting a user-agent string with the token GPTBot/1.0 and a single published IP range, 40.83.2.64/28. Ars Technica reported on Aug 11, 2023 that sites were quickly announcing plans to block it.

Both details have since changed. The documented string is now version 1.4, and the gptbot.json file listed 18 IP prefixes when we checked it on Nov 4, 2026. Treat both as values to look up, not values to hard-code.

Why it matters for B2B and consumer brands

Most brands face one real decision here: what to do about training, separately from search. Allowing OAI-SearchBot is what OpenAI ties to appearing in ChatGPT search answers. GPTBot is a different question about whether your public content may feed future models. OpenAI has not said that allowing GPTBot improves how often a site is cited in ChatGPT, so a training decision should be made on its own terms (licensing, legal position, brand policy) and not as a ranking tactic.

The common mistake is a single rule that blocks every OpenAI agent, or a security tool that blocks them by default. That can remove a site from ChatGPT search answers when the intent was only to opt out of training.

Discovery is the second issue. In a 41-day single-site experiment we covered in Six AI Crawlers Found Zero of 293 Pages, GPTBot and five other non-Google crawlers found none of the 293 pages whose only inbound links were injected by JavaScript. After the links were changed to plain HTML, GPTBot crawled 250 of those pages in under three hours. The test removed sitemaps on purpose, so real sites would see a less absolute result, but plain <a href> links are cheap insurance.

What to do about it

1. Write separate robots.txt groups for GPTBot and OAI-SearchBot. Decide your training policy for GPTBot, and allow OAI-SearchBot if you want to appear in ChatGPT search. Our ChatGPT optimization guide walks through the configuration.
2. Check your CDN, firewall and bot-management settings. A robots.txt allow does nothing if the request is blocked at the edge. OpenAI recommends allowing requests from its published IP ranges for OAI-SearchBot.
3. Verify bots in your server logs. Filter by user-agent token, then confirm the source IP falls within the matching OpenAI JSON file. A request that claims to be GPTBot from an unlisted IP is not one you should count as OpenAI.
4. Look for sections no crawler has requested. Absences are the signal for discovery problems, including JavaScript-only links.
5. Do not rely on robots.txt to control ChatGPT-User. OpenAI says robots.txt rules may not apply to it, so any hard limit has to be set at the network edge.

How other AI companies compare

The same split between training and search exists elsewhere, with different names. Anthropic says ClaudeBot collects web content that could contribute to model training, and that restricting it signals a site's future materials should be excluded (Anthropic). Perplexity says PerplexityBot surfaces and links sites in its search results and is not used to crawl content for AI foundation models (Perplexity). Google's Google-Extended is a robots.txt token with no separate user-agent string, used to manage Gemini training and grounding, and Google says it does not affect inclusion or ranking in Google Search (Google).

Common misconceptions

• "Blocking GPTBot hides us from ChatGPT." OpenAI ties ChatGPT search visibility to OAI-SearchBot, not GPTBot.
• "Allowing GPTBot gets us cited." OpenAI has made no such statement. Citations in ChatGPT search depend on the search crawler and on whether your content answers the question.
• "robots.txt blocks everything OpenAI sends." OpenAI's own documentation says it may not apply to ChatGPT-User.
• "A GPTBot user agent in the logs proves OpenAI visited." Only a match against the published IP ranges does.

Once access is settled, the useful measurement is whether your brand appears in answers to the questions buyers actually ask across AI engines.