Robots.txt Tester & Generator

How it works

What we check

• File presence — Confirms a robots.txt actually exists at the site root and returns a 200. If there's no file, search engines crawl everything by default — usually fine, but you lose the chance to point them at your sitemap and steer them away from junk URLs.
• Site-wide crawl block — Checks for "Disallow: /" under "User-agent: *", which blocks every crawler from the whole site. This single line is the most common way a staging rule accidentally ships to production and de-indexes a site.
• User-agent blocks — Lists every User-agent the file targets. "User-agent: *" applies to all crawlers; named agents (Googlebot, Bingbot) get their own block. Rules are matched to the most specific applicable group, not combined across groups.
• Disallow & Allow paths — Surfaces each Disallow (paths crawlers should skip) and Allow (exceptions inside a disallowed folder). Order doesn't decide precedence — Google applies the most specific, longest matching rule, so an Allow can override a broader Disallow.
• Sitemap reference — Looks for a "Sitemap:" line with the absolute URL of your XML sitemap. It's optional but recommended — it's a direct, crawler-agnostic way to tell search engines where your full URL list lives.
• Crawl-delay directives — Flags any "crawl-delay" lines. Bing and some other bots honor it to throttle request rate, but Google ignores crawl-delay entirely — controlling Googlebot's rate is done in Search Console, not robots.txt.

Common issues we catch

• "Disallow: /" left over from staging — A development site blocks all crawlers to stay out of search. When it goes live, that line ships with it, and the entire site silently drops out of the index. It's the single most damaging robots.txt mistake and the first thing we check for.
• Blocking CSS and JavaScript — Disallowing /wp-includes/, /assets/, or script folders feels tidy, but Google renders pages like a browser. If it can't fetch your CSS and JS, it may see a broken layout and misjudge mobile-friendliness and content — hurting rankings, not helping them.
• Confusing robots.txt with noindex — Disallowing a page does NOT remove it from Google. A blocked URL can still appear in results (as a bare link with no snippet) if other pages link to it. To truly keep a page out, allow crawling and use a noindex meta tag — Google has to crawl it to see the noindex.
• Missing or relative Sitemap path — A "Sitemap:" line must be a full absolute URL (https://example.com/sitemap.xml), not a relative path like /sitemap.xml. A relative path is ignored, so crawlers never get the pointer you meant to give them.
• Relying on crawl-delay for Googlebot — Adding "crawl-delay: 10" to slow Google down does nothing — Google doesn't support the directive. People add it after server-load scares and assume it worked. Use the crawl-rate setting in Search Console instead.
• Wildcard and path-prefix mistakes — Disallow paths are prefix matches: "Disallow: /blog" also blocks /blog-archive. Use "$" to anchor the end ("/blog$") and "*" for wildcards ("/*?*" to block all query strings). Getting these wrong blocks far more — or far less — than intended.
• File not at the exact root — robots.txt only works at the protocol+host root: https://example.com/robots.txt. A file at /folder/robots.txt or on a different subdomain is ignored. Each subdomain (and http vs https) needs its own file at its own root.

Where this matters

• Googlebot — Reads /robots.txt, respects Disallow/Allow with longest-match precedence and wildcard/$ support, reads Sitemap lines — but ignores crawl-delay. A blocked URL can still be indexed if linked elsewhere.
• Bingbot & other major crawlers — Follow the same core syntax and additionally honor crawl-delay to throttle request rate. Most reputable bots respect robots.txt voluntarily; it's a polite request, not an access wall.
• WordPress — Serves a virtual robots.txt by default unless a physical file or SEO plugin overrides it. The 'Discourage search engines' setting injects a site-wide Disallow — the exact switch that de-indexes sites when left on after launch.
• Shopify — Generates robots.txt automatically and historically blocked some store paths by default. It's now editable via the robots.txt.liquid theme template, so custom rules are possible but easy to break.
• Wix, Squarespace & hosted builders — Auto-manage robots.txt with limited editing. Some platforms gate edits behind a paid plan or a built-in SEO panel, so use this tester to confirm what's actually being served versus what you set.