Fetch and analyze the robots.txt file for any domain. See every crawl rule and sitemap reference, get warned if you are accidentally blocking search engines, and download a recommended, SEO-safe robots.txt you can drop straight onto your site.
Fetch and analyze the robots.txt file for any domain. See every crawl rule and sitemap reference, get warned if you are accidentally blocking search engines, and download a recommended, SEO-safe robots.txt you can drop straight onto your site.
This is one of several free SEO tools from Custom Web Audits. For a complete, prioritized analysis of your whole website, run a full audit.