Skip to tool

AI Crawler Policy Generator

Generate robots.txt rules for AI crawlers and bots.

Output will appear here…

How to Use This Tool

  1. Fill in Allow (one per line or comma), and Disallow (one per line or comma). Each control is labeled; you do not need to paste a JSON blob.
  2. Click "Run" to run the tool in your browser.
  3. Read the result below the form and copy it if you need it in another app.

Learn More About AI Crawler Policy Generator

About AI Crawler Policy Generator

Generate robots.txt rules for AI crawlers. Control which AI bots can access your site.

About

Generate robots.txt rules for AI crawlers. Control which AI bots can access your site.
Generate robots.txt rules for AI crawlers. Control which AI bots can access your site.
Runs in browser
Yes
No signup required
Yes

Examples

Block AI bots

Block AI bots
Input
{
  "allow": "",
  "disallow": "GPTBot\nClaudeBot\nPerplexityBot"
}
Output
# AI Crawler Policy
# Generated by AzWebTools

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: *
Allow: /

Allow some

Allow some
Input
{
  "allow": "GoogleBot\nBingBot",
  "disallow": "GPTBot"
}
Output
# AI Crawler Policy
# Generated by AzWebTools

User-agent: GoogleBot
Allow: /

User-agent: BingBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

All blocked

All blocked
Input
{
  "allow": "",
  "disallow": "GPTBot, ClaudeBot, CCbot"
}
Output
# AI Crawler Policy
# Generated by AzWebTools

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCbot
Disallow: /

User-agent: *
Allow: /

Use Cases

  • Check ai crawler policy generator values while you draft a page, spreadsheet, or config.
  • Fill Allow (one per line or comma), Disallow (one per line or comma) and compare the result to a second source.
  • Keep sensitive inputs on-device — this page runs the calculation in the browser.

Frequently Asked Questions

Related Tools

Related tools: