Controlling which AI crawlers can access your site
Major AI companies run named crawlers to gather training data or to fetch pages live when their assistant is asked to browse the web - GPTBot and ChatGPT-User from OpenAI, ClaudeBot from Anthropic, Google-Extended for Gemini, and others. Each one respects the standard robots.txt convention, letting you allow or block them individually rather than blocking all crawlers at once. Choose your preference for each bot and generate a ready-to-publish robots.txt.
Why training crawlers and live-browsing crawlers are listed as separate bots
Many AI companies run two distinct crawlers for two different purposes - one that gathers pages to train future models (like GPTBot), and a separate one that fetches a page live only when a user actually asks the assistant to browse it (like ChatGPT-User). Blocking one doesn't automatically block the other, which is why this tool lists them separately: you might want to opt out of training data collection while still allowing your page to be fetched when a user directly asks an assistant to look at it.
Why blocking an AI crawler doesn't retroactively undo past collection
A robots.txt rule only tells a well-behaved crawler what to do going forward - it has no mechanism to reach back and remove content a crawler already collected during an earlier visit before the rule existed. That's an important distinction to understand before assuming a new block fully addresses content already gathered in the past.
Why blocking these bots doesn't affect your regular Google Search ranking
Googlebot (the crawler that powers standard Search indexing) is a completely separate crawler from Google-Extended (which controls use in Gemini and AI Overviews training) - blocking one has no effect on the other, which means you can opt out of AI-specific uses of your content without any impact on how your site ranks in regular Google Search results.