Test any URL against robots.txt for 20 search and AI bots at once, including GPTBot, ClaudeBot, and PerplexityBot. See the exact rule behind each answer, fix syntax errors, and generate a new file.
Each free tool covers one piece. A free SEO audit covers your technical setup, on-page content, links, and AI visibility together.
Get your free SEO auditSend your site and Furqan will personally review your backlinks, brand mentions, technical and on-page SEO, and AI visibility. You'll hear which service fits, or that you don't need one yet.
Prefer email?
furqan@chungdigital.comEnter a URL to fetch its site's robots.txt, or paste a file. Results update as you edit.
Example file for a made-up SaaS site. Enter a URL, or paste your own robots.txt below.
Testing path /blog/drafts/launch-notes
GPTBot (OpenAI)
TrainingBlocked
Line 11: Disallow: /
Collects pages that may be used to train OpenAI's models.
Search engines
4 of 4 allowed
AI search
3 of 4 allowed
AI training
6 of 8 allowed
User actions
4 of 4 allowed
Pick a bot to see the rule that decides it, highlighted in the file.
Highlighted: the group GPTBot follows, and the deciding rule.
One URL or path per line, up to 50.
| Path | Result | Rule |
|---|---|---|
| / | PassedAllowed | No matching rule |
| /pricing | PassedAllowed | No matching rule |
| /app/settings | ErrorBlocked | Line 3: Disallow: /app/ |
| /api/docs/start | PassedAllowed | Line 6: Allow: /api/docs/ |
| /blog/drafts/launch-notes | PassedAllowed | No matching rule |
Chung Digital AI
Get a plain review of what this file allows and blocks, and what to change. It reads the bot results and warning types listed below, not your file itself.
What it reads
Ask for a plain review of this robots.txt.
Pick a starting point and an AI bot policy, adjust the paths, and copy the file. Then test it above before you upload it.
# robots.txt made with the Chung Digital robots.txt generator User-agent: * Disallow: /app/ Disallow: /account/ Disallow: /api/ Disallow: /search # AI bots blocked from the whole site User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: meta-externalagent User-agent: Amazonbot User-agent: Bytespider Disallow: / Sitemap: https://www.example.com/sitemap.xml
Upload it to the root of your domain, so it loads at /robots.txt.
Enter any URL and we fetch that site's robots.txt, then test the URL against every rule for more than 20 search and AI crawlers at once. You can also paste a file, or edit the fetched one, and every result updates as you type.
Each bot shows whether it may crawl the URL and which line decided it, so you can see exactly why a page is blocked. The validator flags lines crawlers will skip, rules Google ignores, and a missing or broken sitemap line.
AI companies run different bots for different jobs. Training bots such as GPTBot and ClaudeBot collect pages for model training. Search bots such as OAI-SearchBot and Claude-SearchBot index pages so assistants can cite them. User-action bots open a page when someone asks an assistant to read it.
That split lets you choose. Many SaaS sites block training bots but allow AI search bots, so they still show up in AI answers. Blocking a training bot does not remove you from that company's AI search.
| User-agent token | Operator | Type | What it does |
|---|---|---|---|
| Googlebot | Search engines | Crawls pages for Google Search, including AI Overviews. | |
| Bingbot | Microsoft | Search engines | Crawls pages for Bing, which also feeds Microsoft Copilot answers. |
| Applebot | Apple | Search engines | Crawls pages for Siri, Spotlight, and Safari suggestions. |
| DuckDuckBot | DuckDuckGo | Search engines | Crawls pages for DuckDuckGo search. |
| OAI-SearchBot | OpenAI | AI search | Finds pages to show and link in ChatGPT search results. |
| Claude-SearchBot | Anthropic | AI search | Indexes pages to improve search results in Claude. |
| PerplexityBot | Perplexity | AI search | Indexes pages for Perplexity's answers and citations. |
| DuckAssistBot | DuckDuckGo | AI search | Fetches pages for DuckDuckGo's AI-assisted answers. |
| GPTBot | OpenAI | AI training | Collects pages that may be used to train OpenAI's models. |
| ClaudeBot | Anthropic | AI training | Collects public pages that may be used to train Anthropic's models. |
| Google-Extended | AI training | A token, not a crawler: controls whether Gemini models may train on your pages. It doesn't affect Google Search. | |
| Applebot-Extended | Apple | AI training | A token, not a crawler: controls whether Apple may use your pages to train its AI models. |
| CCBot | Common Crawl | AI training | Builds the open Common Crawl dataset, which many AI models are trained on. |
| meta-externalagent | Meta | AI training | Crawls pages for uses such as training Meta's AI models. |
| Amazonbot | Amazon | AI training | Crawls pages to improve Amazon's services, which can include training its AI models. |
| Bytespider | ByteDance | AI training | ByteDance's crawler, commonly listed as an AI data crawler. |
| ChatGPT-User | OpenAI | User actions | Opens a page when a ChatGPT user or custom GPT asks for it. |
| Claude-User | Anthropic | User actions | Opens a page when a Claude user asks for it. |
| Perplexity-User | Perplexity | User actions | Opens a page when a Perplexity user asks for it. |
| MistralAI-User | Mistral AI | User actions | Opens a page when a Le Chat user asks for it. |
Google and most major crawlers follow the same three rules, set out in the robots.txt standard (RFC 9309).
A crawler obeys the group that names it, such as User-agent: GPTBot, and ignores the * group. Only bots with no group of their own follow *.
Inside the group, the rule with the longest matching path decides. Allow: /api/docs/ beats Disallow: /api/ for /api/docs/start.
When an Allow and a Disallow rule match with the same length, Allow wins. * matches any characters, and $ marks the end of the URL.
"Blocked by robots.txt" in Google Search Console means a Disallow rule stops Googlebot from crawling the URL. Google can still index the URL from links, but without reading it, so it may appear with no description.
Paste the URL above to find the rule responsible. If the page should be in search, remove or narrow that rule. If it shouldn't be, allow crawling and add a noindex tag instead, because Google has to crawl a page to see noindex.
Add one Sitemap line per sitemap, anywhere in the file. It must be a full URL including https://. Crawlers read it no matter which User-agent group it sits near.
A sitemap line doesn't replace submitting the sitemap in Google Search Console, but it helps every other crawler, including AI search bots, find your pages.
User-agent: * Disallow: /app/ Sitemap: https://www.example.com/sitemap.xml Sitemap: https://www.example.com/blog/sitemap.xml
Letting AI search bots in is step one. Our AI brand mentions service works on getting your brand cited in AI answers, and our SaaS SEO services cover technical SEO like robots.txt across your site. Or request a free SEO audit.
It always sits at the root of the domain: add /robots.txt after the domain name, for example https://www.example.com/robots.txt. Each subdomain has its own file. You can also enter any page URL above and this tester finds the file for you.
No. robots.txt controls crawling, not indexing. A blocked URL can still appear in Google if other pages link to it. To keep a page out of search, allow crawling and use a noindex tag.
ClaudeBot is Anthropic's web crawler. It collects public pages that may be used to train Anthropic's Claude models. Anthropic also runs Claude-SearchBot for search results in Claude and Claude-User for pages a user asks Claude to open, and each one can be allowed or blocked separately.
It depends on your goal. GPTBot collects data for OpenAI's model training. Blocking it opts you out of training, but it doesn't affect OAI-SearchBot, which decides whether your pages can appear in ChatGPT search. Many SaaS sites block GPTBot and allow OAI-SearchBot.
No. Google-Extended only controls whether Gemini models may use your pages for training and grounding. Google Search, including AI Overviews, uses Googlebot, so blocking Google-Extended doesn't change how you rank.
No. Google ignores Crawl-delay. Bing and Yandex read it. If Googlebot crawls too fast, your server can reply with a 429 or 503 status to slow it down.
On WordPress, an SEO plugin usually has a robots.txt editor, or you can upload a file to the site root. On Shopify, edit the robots.txt.liquid template in your theme, because Shopify generates the file for you.
Yes. We save the URL you fetch, the robots.txt you test, the paths you check, and any error for 12 months, with your IP address and approximate location, so we can find what goes wrong and improve the tester. If you use the AI review, the bot results and warning types are sent to our AI provider, not your file. See our privacy policy.