Why robots.txt alone does not tell the whole story
AI bots fall into different categories, and robots.txt does not control all of them the same way. Training crawlers like GPTBot and ClaudeBot generally respect a Disallow rule. Bots that fetch a page because a specific user asked an assistant to look at it are a different story: Google’s fetchers ignore robots.txt entirely by design, and Perplexity has been documented running crawlers that skip it too.
This tool reads your robots.txt and checks it against the AI crawlers we track, split into the categories that actually matter: training, search and retrieval, user-triggered fetchers, and the opt-out tokens that are not crawlers at all. Paste your file below to see where you stand.
Paste a robots.txt above and click Check access to see results here.
How we calculate this
Matching: we look for an exact User-agent block matching each bot name, then fall back to the wildcard User-agent: * block if the bot is not named directly. A Disallow: / blocks everything, a narrower Disallow blocks part of the site, and no matching rule means the bot is allowed by default.
Important caveat: respecting robots.txt is decided by each vendor, not guaranteed by the standard. Google’s user-triggered fetchers ignore robots.txt entirely by their own documentation. Perplexity states that it respects robots.txt, but independent research from Cloudflare found it running undeclared crawlers that do not. Bytespider has no official vendor documentation at all. A “Blocked” result here means your file says so, not that the bot will necessarily comply.
This is a directional check based on published vendor policies, not a certified robots.txt parser or a guarantee of compliance. Bot behaviour can change without notice.