Create, test, and download robots.txt files with an intuitive visual builder. Block AI bots, set crawl delays, and validate your configuration instantly.
Presets
User-Agent #1
robots.txt Preview
User-agent: *Allow: /
β No issues found. Your robots.txt looks good!
Test URL
What robots.txt Does β and Doesn't Do
Its role (RFC 9309)
`robots.txt` at your site root asks crawlers not to crawl certain paths β useful for admin areas, internal search pages, and crawl-budget management.
The big misconception: blocking β de-indexing
A URL blocked in robots.txt can still appear in search results if other sites link to it. To remove a page from the index, use a `noindex` meta tag β and crucially, don't block that page in robots.txt, or crawlers can never see the noindex.
Practical checklist
`Disallow: /` blocks the entire site. Staging configs shipping to production is a real and common accident.
Include a `Sitemap:` line pointing to your sitemap.
Blocking AI crawlers (GPTBot, ClaudeBot, PerplexityBot) is a choice: it protects content from training but also removes you from AI search citations.
Frequently Asked Questions
A robots.txt file is a plain text file placed at the root of a website (e.g., example.com/robots.txt) that tells web crawlers which pages or sections they are allowed or not allowed to access. It follows the Robots Exclusion Protocol standard.
Use this generator to visually build your robots.txt rules. Add user-agent blocks, set Allow/Disallow directives for specific paths, optionally add a crawl-delay, and include your sitemap URLs. Then download or copy the generated file and upload it to your website's root directory.
Use the 'Block AI Bots' preset, or manually add User-agent blocks for GPTBot, ChatGPT-User, CCBot, Google-Extended, ClaudeBot, anthropic-ai, Bytespider, and other AI crawlers, each with 'Disallow: /' to block them from crawling your entire site.
'Disallow: /' blocks the specified user-agent from accessing any page on your site. The forward slash (/) represents the root of your site and all paths beneath it.
Crawl-delay is a directive that tells crawlers to wait a specified number of seconds between successive requests. Not all crawlers honor it β Google ignores Crawl-delay (use Google Search Console instead), but Bing and Yandex support it.
Yes. Adding a 'Sitemap:' directive pointing to your XML sitemap helps search engines discover all your pages more efficiently. You can include multiple sitemap URLs.
Use the URL tester below the preview. Enter a URL path and select a user-agent to check whether the path would be allowed or blocked based on your current rules.
'Disallow' prevents a crawler from accessing the specified path. 'Allow' explicitly permits access to a path, which is useful for overriding a broader Disallow rule. When rules conflict, the most specific (longest) path match takes precedence.
No. robots.txt is a guideline, not a security mechanism. Well-behaved crawlers will respect it, but malicious bots can ignore it. For truly restricting access, use server-side authentication or password protection.
The wildcard user-agent (*) applies rules to all crawlers that don't have their own specific block. It's a catch-all default. If a specific user-agent block exists (e.g., Googlebot), that bot will follow its own block instead of the wildcard.