AI Crawler Robots.txt Generator
Choose which AI crawlers may read your site – for search answers or model training – and get ready-made robots.txt rules.
Free, no sign-up. Runs in your browser – nothing you type is sent to our servers.
Our editors review it and distribute it to news sites in the USA, UK, UAE, India and beyond – with a report of every live link.
About this tool
AI companies use different crawlers for different jobs: some fetch pages to answer a user’s question or show search results, others collect data for training models. Blocking search crawlers can keep you out of AI answers, so decide bot by bot.
Frequently asked questions
Will blocking AI crawlers remove me from ChatGPT or Perplexity answers?
Blocking their search crawlers (such as OAI-SearchBot or PerplexityBot) can stop your pages being used in their answers. Blocking training crawlers (such as GPTBot or CCBot) does not affect search answers.
Do all bots obey robots.txt?
Reputable crawlers do; it is a request, not a technical barrier.
Send on WhatsApp