Robots.txt Tester
Paste your robots.txt and a URL path to see whether Googlebot, Bingbot and AI crawlers such as GPTBot and ClaudeBot are allowed – and which rule decides.
Free, no sign-up. Runs in your browser – nothing you type is sent to our servers.
Our editors review it and distribute it to news sites in the USA, UK, UAE, India and beyond – with a report of every live link.
About this tool
The tester follows Google’s rules: a crawler uses the most specific group that names it, the longest matching rule wins, and “Allow” wins a tie. Wildcards (*) and end anchors ($) are supported.
Frequently asked questions
Does robots.txt stop a page from being indexed?
No. It controls crawling. To keep a page out of search results use a noindex tag and let the page be crawled.
Which bots are AI crawlers?
Examples include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended and CCBot.
Send on WhatsApp