robots.txt Generator

Why your robots.txt now matters for AI

For twenty years robots.txt only had to speak to search engine spiders. Now the same file is where you tell GPTBot, ClaudeBot, PerplexityBot and a dozen others whether they may read your site for training or answers, and most sites have never made that choice deliberately.

Set each crawler to allow or block below and copy the result. If you would rather audit a file you already have, the AI Crawler Access Checker does the reverse job.