Robots.txt generator

Pick which bots you let in, set your sitemap, and download a copy-ready robots.txt file.

Default policy

Turn this on to fully block indexing. Leave off for the standard"allow all, override per-bot" behavior.

Sitemap

Crawl-delay (optional)

Google ignores this. Bing and a handful of smaller bots honor it.

Per-bot overrides

  • Googlebot
  • Bingbot
  • GPTBot (OpenAI)
  • ClaudeBot (Anthropic)
  • CCBot (Common Crawl)
  • PerplexityBot
  • Applebot-Extended

Custom rules (advanced)

One rule per line. Appended verbatim — e.g. Disallow: /admin or Allow: /public/.

robots.txt

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Annie watches your robots.txt weekly

Accidentally block GPTBot? Lose a referral? Annie flags it before you do. Free for 7 days.

Get Annie — free for 7 daysTry another tool