SEO / Webmaster
Robots.txt Generator
Build a robots.txt file by picking a preset or composing rules yourself. Add sitemap URLs and an optional crawl delay; download when done.
Quick presets
User-agent rules
Google ignores Crawl-delay; Bing and Yandex respect it.
Generated robots.txt
Frequently asked questions
Where do I put the generated robots.txt file?
Copy or download the output and place it at the root of your site, so it is reachable at https://yourdomain.com/robots.txt — crawlers only look for it there.
What do the presets do?
They fill in common rule sets for you — Allow all, Block all, a Standard setup, a WordPress configuration, and one that blocks common AI scrapers — which you can then edit per user-agent group.
Does the Crawl-delay setting work for Google?
No. Google ignores the Crawl-delay directive (control crawl rate in Search Console instead), but Bing and Yandex do respect it.
Does robots.txt keep a page out of Google entirely?
Not reliably. Disallow only asks crawlers not to fetch a page; to keep a URL out of search results use a noindex meta tag or password protection instead.