Robots.txt & XML Sitemap Checker
Check a live site’s robots.txt and XML sitemaps in one go: which crawlers can reach a page, which rule decides, how many URLs the sitemaps list and whether a sample of them returns 200.
Free, no sign-up. Our server fetches the public page you enter – nothing is stored.
Our editors review it and distribute it to news sites in the USA, UK, UAE, India and beyond – with a report of every live link.
About this tool
The checker downloads /robots.txt from the site root, groups the rules by user-agent and applies Google’s longest-match logic to the path you entered – for Googlebot, Bingbot and the main AI crawlers. It then reads every Sitemap: line (or tries /sitemap.xml), follows sitemap index files and samples up to 30 listed URLs to make sure they load.
Want to try out rules before you publish them? Use the robots.txt tester to paste a draft, and the XML sitemap generator to build a sitemap from a list of URLs.
Frequently asked questions
Does robots.txt keep a page out of Google?
No. It controls crawling, not indexing. A blocked URL can still appear in results if other sites link to it. Use a noindex tag – on a crawlable page – to keep it out.
What should a sitemap contain?
Only canonical URLs that return 200 and that you want indexed. Leave out redirects, error pages, noindex pages and duplicates.
Why is lastmod important?
An accurate lastmod date tells search engines and AI crawlers which pages changed, so they re-fetch them sooner.
Send on WhatsApp