Robots.txt Generator

The Robots.txt Generator builds a syntactically correct robots.txt file from a preset or custom set of Allow/Disallow rules and an optional sitemap reference, and includes a separate validator that checks a pasted robots.txt for common structural mistakes — a directive with no colon, a Disallow rule listed before any User-agent line, an unrecognized directive name. When you need to create robots.txt from scratch, or run a pasted file through the built-in robots.txt validator to confirm the robots.txt syntax is clean, this covers both directions.

It's used when launching a new site and needing a starting robots.txt that doesn't accidentally block the whole site from search engines (a single misplaced 'Disallow: /' has taken real sites out of Google's index before), when blocking a staging environment, admin panel or internal search results page from being crawled, and when auditing an existing robots.txt that a previous developer wrote by hand to check it doesn't contain a silent syntax error. The three presets — allow all, block all, block admin-only paths — cover the most common starting points; a fourth custom mode accepts arbitrary Allow and Disallow path lists plus an optional crawl-delay.

robots.txt is a plain-text convention, not an enforced security boundary — well-behaved crawlers like Googlebot respect it, but it doesn't password-protect anything, and disallowing a path doesn't remove an already-indexed page from search results (that requires a noindex meta tag or Search Console removal request instead). The validator is a basic line-by-line syntax check against the standard directive set (User-agent, Allow, Disallow, Sitemap, Crawl-delay, Host) — it catches structural mistakes, not semantic ones like blocking a page you actually wanted crawled. Everything runs in your browser; nothing about your site is sent anywhere by using this tool.