AI Crawler Access Checker
See whether ChatGPT, Claude, Perplexity, Gemini, Copilot and 15 other AI crawlers can reach a page – through robots.txt, page-level directives and your firewall or CDN.
Free, no sign-up. Our server fetches the public page you enter – nothing is stored.
Our editors review it and distribute it to news sites in the USA, UK, UAE, India and beyond – with a report of every live link.
About this tool
AI assistants use different bots for different jobs. Search and answer bots (such as OAI-SearchBot, Claude-SearchBot and PerplexityBot) fetch pages so they can be cited in answers; training crawlers (such as GPTBot, ClaudeBot and CCBot) collect data to train models. Blocking the first group removes you from AI answers; blocking the second is a business choice.
The checker evaluates your robots.txt for 22 AI user-agents, reads noindex, nosnippet and similar directives on the page, requests the page with real AI user-agent strings to catch firewall or CDN blocks, and looks for an llms.txt file. It finishes with a ready-to-paste robots.txt block. To write rules from scratch, try the AI crawler robots.txt generator.
Frequently asked questions
Will blocking GPTBot remove me from ChatGPT answers?
Not by itself. GPTBot is OpenAI’s training crawler; ChatGPT search uses OAI-SearchBot and ChatGPT-User. Block or allow each one separately.
Why does robots.txt allow a bot but the live test fails?
Your server, firewall or CDN (for example a bot-management rule) may be returning an error to that user-agent even though robots.txt permits it.
Does Google-Extended affect Google Search?
No. Google-Extended only controls use in Gemini models; Googlebot handles Search, including AI Overviews.
Send on WhatsApp