FindSiteRank

Robots.txt Checker

Your robots.txt file tells search engine crawlers which parts of your site they can and cannot access. A misconfigured file can accidentally block Google from indexing important pages. Enter a domain to fetch and analyze the file instantly.

Understanding robots.txt

The robots.txt file sits at the root of your domain (e.g. example.com/robots.txt) and uses simple directives to communicate with bots. Here's what each directive means:

  • User-agent — specifies which crawler the rules apply to. * means all crawlers.
  • Disallow — tells crawlers not to access the specified path. Disallow: / blocks the entire site.
  • Allow — explicitly permits a path, even if a parent directory is disallowed.
  • Sitemap — points crawlers to your sitemap.xml for faster and more complete indexing.