robots.txt & Sitemap Checker

See what search engines are allowed to crawl, and whether your sitemap is present and readable.

GO DEEPERWhat do you want to uncover next?
🤖 See exactly what crawlers can access🗺️ Check DNS before blaming indexing🔎 Inspect page metadata

Terms explained

robots.txtSearch EngineSitemapWeb CrawlerGoogle Search ConsoleIndexing

Frequently asked questions

What does "Disallow: /" do?

It tells all crawlers not to crawl the entire site. It is a common cause of pages disappearing from Google after a staging site goes live.

Where should my sitemap be?

At /sitemap.xml, and referenced in robots.txt with a Sitemap: line.