AI crawler check
Enter a domain — we read the real robots.txt and show which AI bots are allowed and which are blocked. Free, no signup.
Bots we check
The list is built from public vendor documentation. When a bot’s status isn’t backed by its own vendor docs, we say so explicitly.
AI crawlers explained
Training, AI search, user requests — and ready robots.txt blocks
llms.txt generator
Build the file from the real pages in your sitemap
llms.txt validator
Check an existing file against the proposed format
Platform guides: