WEBSITE TEMPLE / SEO TOOLS

ROBOTS.TXT GENERATOR

Build a clean robots.txt with Allow, Disallow, and Sitemap rules, plus separate controls for how AI crawlers access the site.

No signup. Generated locally.

None of the rules you enter are sent to a server. The finished robots.txt is downloaded directly from your browser.

DomainPrimary User-agent

Blocked pathsOne path per line. Leave empty if you do not want to block anything.Allowed exceptionsUse Allow only when you need to permit a more specific path inside a blocked section.

Sitemap URL

AI & SEARCH CRAWLERS

Add a separate rule only when a specific crawler should behave differently from the primary rules.

ChatGPT Search

OAI-SearchBot

Visibility in ChatGPT Search

OpenAI training

GPTBot

Potential model-training use

Claude Search

Claude-SearchBot

Visibility in Claude Search

Claude training

ClaudeBot

Potential model-training use

Gemini / Google AI

Google-Extended

AI use; does not affect Google Search

Perplexity Search

PerplexityBot

Visibility in Perplexity Search

Grok / xAI

No officially confirmed robots.txt token

We do not generate a guessed token. If xAI publishes an official crawler token, it can be added here.

LIVE PREVIEW

User-agent: *
Disallow:

CHECK THE PUBLISHED FILE

Test the live robots.txt

After uploading the file to the site root, test a specific URL with Googlebot, ChatGPT, Claude, Gemini, or another crawler.

OPEN ROBOTS.TXT TESTER

robots.txt controls crawling, not privacy.

Do not use robots.txt to hide sensitive pages. Sensitive content needs real access control.

SEO & CRAWLING

Create robots.txt with separate controls for search and AI crawlers

robots.txt tells crawlers which parts of a site may be crawled. The generator supports User-agent, Allow, Disallow, and Sitemap directives, plus separate controls for ChatGPT Search, OpenAI training, Claude, Gemini, and Perplexity.

How to create robots.txt

Set the primary crawl rules for all crawlers or a specific User-agent.

Add separate AI crawler rules only when a crawler should behave differently from the primary rules.

Copy or download the file, publish it at the site root, and verify the live version with Robots.txt Tester.

Important rules

The Sitemap directive should use a fully qualified absolute URL.

Google-Extended controls certain AI uses of content and does not affect Google Search.

robots.txt controls crawling, not access to sensitive content.

Robots.txt FAQ

Where should robots.txt be hosted?

The file should be available at the root of the specific host, for example https://example.com/robots.txt.

Can I manage ChatGPT and Claude crawlers separately?

Yes. Official crawler tokens can have their own User-agent groups. The generator adds a separate rule only when you explicitly choose one.

Can robots.txt hide sensitive content?

No. Use real access control for sensitive content. robots.txt controls crawling, not security.

SEO tools

Robots.txt Tester Sitemap Generator