HECTOR
Free interactive tool · 2 minutes

Write a robots.txt for the AI-crawler era

Pick which AI crawlers get through the door and which get turned away — plus an optional Sitemap line and paths to keep out of every bot. You get copy-paste robots.txt text in seconds. No email, nothing stored. Checked your site with our AI Crawler Access Checker first? This is the file that fixes what it found.

Only used to build the Sitemap line below. The site is never fetched.

AI crawlers — allow or block?
GPTBot OpenAI\u2019s main training crawler.
ChatGPT-User Powers ChatGPT\u2019s live browsing and link-checking.
CCBot Common Crawl — trains many open and commercial models.
anthropic-ai Anthropic\u2019s primary crawler, feeding Claude.
ClaudeBot Claude\u2019s dedicated agent.
PerplexityBot Perplexity\u2019s retrieval bot — block this to be cited less.
Google-Extended Google\u2019s AI-training crawler (separate from Googlebot).
Bytespider ByteDance\u2019s AI crawler.
meta-externalagent Meta\u2019s AI data crawler.

e.g. admin pages, login pages, staging folders.