Does the tester change robots.txt?
No. It only reads the public file and evaluates the selected URL and crawler.
WEBSITE TEMPLE / SEO TOOLS
ROBOTS.TXT TESTER
Test a website’s live robots.txt and see whether a specific URL is accessible to Google, ChatGPT, Claude, Gemini, Perplexity, or another crawler.
Test the live file. No signup.
We fetch the public robots.txt only for the check. We do not store the file or the URL you enter.
LIVE ROBOTS.TXT CHECK
Check whether a URL is allowed for Google and AI crawlers
Robots.txt Tester fetches the public robots.txt file and applies its rules to a specific URL and crawler, so you see the live configuration rather than a local preview.
How the test works
01
Enter a URL from the website you want to test.
02
Choose a crawler such as Googlebot, OAI-SearchBot, Claude-SearchBot, Google-Extended, or PerplexityBot.
03
See whether the URL is allowed or blocked and which Allow/Disallow rule matched.
When the tester is useful
After editing robots.txt, to verify the file that is actually live.
When different crawlers should have different access to the same URL.
When a sitemap or SEO audit reveals a crawlability issue.
Robots.txt Tester FAQ
What if robots.txt is missing?
We show that the file was not found. A missing robots.txt file does not itself block crawling.
Does it support AI crawlers?
Yes. It includes documented crawler tokens from OpenAI, Anthropic, Google, and Perplexity, plus a custom User-agent option.
SEO tools