Should I Block GPTBot? The Real Decision Behind the Robots.txt Line

The short answer

For most independent sites, blocking GPTBot is a small and reversible decision. It stops OpenAI from using your pages to train future models. It does not, and this is the part people get wrong, take you out of ChatGPT's search answers. Those run on a different crawler with a different name, and they are the ones that can actually send you a visitor.

So the honest answer to whether you should block it is: sure, if you object to model training, go ahead. Just do it on purpose, and do not assume it protects you from anything else or that it will change your traffic. It will not do either.

Two jobs, two kinds of crawler

Every major AI company now runs more than one bot, and they do genuinely different jobs. Lumping them together is where the bad advice starts. There are training crawlers, which read your site to help build the model itself, and there are retrieval crawlers, which read your site so an assistant can quote you live and link back. The first kind gives you nothing you can measure. The second kind can send a human to your door.

• GPTBot is OpenAI's training crawler. Disallowing it in robots.txt tells OpenAI your content should not be used to train its foundation models. It sends no traffic and produces no citation.
• CCBot is Common Crawl's crawler. Its open dataset has been used to train GPT, Llama, Mistral and many other models, so it is training infrastructure under another name. Nothing comes back to you directly.
• Google-Extended is not even a separate crawler. It is a control token in robots.txt. Disallowing it opts your content out of training Gemini and Vertex AI, and Google's own documentation says it does not affect your Search ranking.
• ClaudeBot is Anthropic's training crawler. Same category as GPTBot: block it to stay out of training, and expect no traffic in return.
• OAI-SearchBot is the crawler behind ChatGPT search. This is the one that decides whether ChatGPT can surface your page with a link. Retrieval, not training.
• PerplexityBot indexes pages for Perplexity's answers and citations. Perplexity's own documentation says it does not build foundation models, so this is about being quoted, not about being trained on.
• ChatGPT-User and Claude-User are live fetches. When a person asks and the assistant opens a page to answer, that is these agents at work. OpenAI notes that because a user starts each request, robots.txt rules may not even apply.

Blocking GPTBot does not block ChatGPT search

This is the mistake that quietly costs people the most. They read that GPTBot is the OpenAI bot, add one Disallow line, and assume they have opted out of ChatGPT entirely. They have not. They have opted out of training and left search completely untouched, because search runs on OAI-SearchBot.

A webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models.

That is straight from OpenAI's own bot documentation. The two are independent by design. If you want to be cited in ChatGPT search while staying out of the training set, you allow OAI-SearchBot and disallow GPTBot. The far more damaging error runs the other way: catching OAI-SearchBot in a broad blanket rule and vanishing from ChatGPT answers while you thought you were only touching training.

What blocking actually buys you

Strip away the noise and both the cost and the benefit are small, which is why I get annoyed at how much anxiety this topic generates.

Blocking a training crawler buys you one thing: your content is less likely to end up in the next model's training data. That is a real preference and a legitimate one, especially if you publish original research or paid work. What it does not buy you is any measurable outcome. You cannot open your analytics and attribute a signup to being in a training set. There is no referral, no click, no line item.

Keeping a retrieval crawler open buys you the opposite: a shot at being the source an assistant quotes, with a link a real person can follow. That click lands in your analytics like any other referral. It is the only part of this whole conversation you can actually measure.

How I set it up for Conclick

I run Conclick, so here is exactly what I do rather than a generic recommendation. My robots.txt allows the retrieval crawlers: OAI-SearchBot, PerplexityBot, Claude-SearchBot, and the user-triggered fetchers. It disallows the training crawlers: GPTBot, CCBot, ClaudeBot, and Google-Extended. Retrieval open, training closed.

That is a values call, not a growth hack. I would rather my writing help answer a real question and send me the reader than sit silently inside a training corpus I get nothing back from. If your view on training is different, flip those lines without guilt. The retrieval lines are the ones I would never touch, because those are the ones that show up in the numbers.

Measure what retrieval actually sends

The reason to keep retrieval open is that it produces traffic you can see. When ChatGPT, Perplexity or Claude cites you, some readers click the link, and that arrives as referral traffic in your analytics. If you are not watching for it, you will underrate the entire channel and maybe block the wrong bot to protect yourself from a threat that was never there.

Treat AI assistants as a referral source like any other. Watch which pages get cited, how those visitors behave once they land, and whether they do anything worth money. That is ordinary attribution work applied to a new channel, and it is the only way to know whether allowing these crawlers is paying off for you specifically.

For a bootstrapped founder with a few hundred visitors a month, this is not a big project. You already have the data. You just need to name AI referrals as a source and follow those sessions through to whatever counts as a conversion for you. Do that for a quarter and you will have a real answer instead of a vibe.