Esiasetukset:
(Dokumentaatio englanniksi)
What it does
Build a standards-compliant robots.txt with per-user-agent allow/disallow rules, crawl-delay, and sitemap entries. Stack multiple user-agents in one group (the compact User-agent: block the spec defines), add separate groups per bot, and watch the file render live beside the controls. 100% client-side.
How to use it
- Presets — start from Allow all, Block /admin/, or Block AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, CCBot, Google-Extended, Bytespider, PerplexityBot in one stacked group).
- Paste an existing robots.txt — expand the import panel, paste your current file, and Parse & load brings it into the editor (comments are stripped, rules merge per agent).
- Per group: set the User-agents (comma-separated), the Disallow / Allow paths (one per line), and optionally a Crawl-delay.
- Add Sitemap URLs (one per line).
- Copy or Download
robots.txtand drop it at your site root.
Examples
Disallow /admin/ and /private for all bots, link a sitemap →
User-agent: *
Disallow: /admin/
Disallow: /private
Sitemap: https://example.com/sitemap.xmlOne stacked group blocking the AI crawlers →
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
Disallow: /Good to know
- Live validation — the tool flags duplicate user-agents (crawlers merge them), paths missing a leading
/, non-absolute sitemap URLs, and notes that Google has never supportedCrawl-delay(Bing and Yandex honor it). - Shareable — your whole config is encoded in the page URL; copy the address to hand someone the exact setup.
- Private: generation is local.
- An empty disallow list emits
Disallow:(allow all) for that group. - Related tools: Slugify, JSON-LD Structured Data.