robots.txt-generator

(Documentatie in het Engels)

What it does

Build a standards-compliant robots.txt with per-user-agent allow/disallow rules, crawl-delay, and sitemap entries. Stack multiple user-agents in one group (the compact User-agent: block the spec defines), add separate groups per bot, and watch the file render live beside the controls. 100% client-side.

How to use it

1. Presets — start from Allow all, Block /admin/, or Block AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, CCBot, Google-Extended, Bytespider, PerplexityBot in one stacked group).
2. Paste an existing robots.txt — expand the import panel, paste your current file, and Parse & load brings it into the editor (comments are stripped, rules merge per agent).
3. Per group: set the User-agents (comma-separated), the Disallow / Allow paths (one per line), and optionally a Crawl-delay.
4. Add Sitemap URLs (one per line).
5. Copy or Download robots.txt and drop it at your site root.

Examples

Disallow /admin/ and /private for all bots, link a sitemap →

User-agent: *
Disallow: /admin/
Disallow: /private

Sitemap: https://example.com/sitemap.xml

One stacked group blocking the AI crawlers →

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
Disallow: /

Good to know

• Live validation — the tool flags duplicate user-agents (crawlers merge them), paths missing a leading /, non-absolute sitemap URLs, and notes that Google has never supported Crawl-delay (Bing and Yandex honor it).
• Shareable — your whole config is encoded in the page URL; copy the address to hand someone the exact setup.
• Private: generation is local.
• An empty disallow list emits Disallow: (allow all) for that group.
• Related tools: Slugify, JSON-LD Structured Data.