18 crawlers. One verdict each.
Every AI crawler that matters, evaluated against the exact URL you enter, with the rule that decided it.
- OAI-SearchBotOpenAIsearch
- Claude-UserAnthropicbrowse
- PerplexityBotPerplexitysearch
- Google-ExtendedGoogletrain
18 AI crawlers
0 ptsOpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, Meta, DuckDuckGo, Mistral, ByteDance and Common Crawl, grouped into AI search, live browsing and model training.
Real robots.txt evaluation
0 ptsParses every group and applies the standard: the most specific user-agent group wins, the longest matching path wins, wildcards and $ anchors are honoured.
- User-agent: OAI-SearchBotAllow /
- User-agent: ClaudeBotAllow /
- User-agent: PerplexityBotAllow /
- User-agent: GPTBotDisallow /
Verdict for the exact URL
0 ptsAllowed, restricted (blocked on this path but not site-wide) or blocked, with the deciding rule printed next to every crawler.
Page-level directives
0 ptsReads the robots meta tag and the X-Robots-Tag header for noindex, noai and noimageai, which override robots.txt for the page itself.
Ready-to-paste fix
0 ptsGenerates the robots.txt lines that reopen the door for every AI search or browsing crawler you are currently blocking.
Sitemap declarations
0 ptsLists the Sitemap: lines in robots.txt so you can confirm crawlers are pointed at your URL inventory.
Blocking the wrong bot is silent.
Until the traffic is gone.
Most sites that block AI crawlers did it once, in 2023, with a rule that now removes them from ChatGPT search, Perplexity and Claude. The crawlers have changed. The rules have not.
Search crawlers are not training crawlers
OpenAI uses GPTBot for training but OAI-SearchBot for ChatGPT search. Anthropic splits ClaudeBot from Claude-SearchBot. Blocking the training bot is fine. Blocking the search bot makes you invisible in answers.
One wildcard rule can block everyone
A Disallow under User-agent: * applies to every crawler without its own group, including bots launched after you wrote the file. The checker shows which rule actually applies to each bot.
The page can override the file
Even with a permissive robots.txt, a noindex meta tag or an X-Robots-Tag header keeps a page out of the index that AI answers draw from. Both are checked for the URL you enter.
Open the door in order. Search crawlers first, training crawlers on your terms.
- 1
Allow the AI search crawlers explicitly
Add groups for OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User with Allow: /. Explicit groups win over the wildcard group.
- 2
Decide on training crawlers separately
GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent and Bytespider only collect training data. Block them if you wish; it does not affect AI search visibility.
- 3
Keep Googlebot and Bingbot open
Google AI Overviews and Bing Copilot use the classic search crawlers. A block here removes you from both traditional and AI results.
- 4
Check page-level directives
Remove noindex from pages you want cited, and drop noai unless you deliberately opt out of AI answers.
- 5
Re-test after every deploy
robots.txt is easy to overwrite from a staging config. Paste the URL again after each release; results refresh within a few hours.
Open the door, then give them something to cite
SiteLift publishes AI-ready articles with structured data and FAQs on your site, and distributes them so the crawlers you just unblocked have a reason to come back.
No credit card required
Frequently Asked Questions
Everything you need to know about the ai bot access checker.
More free tools
AI Readiness Checker
Check whether AI crawlers can read your site, whether you publish llms.txt, and how well your content is structured to be cited by AI assistants.
llms.txt Generator
Generate a ready-to-publish llms.txt from any website: sitemap discovery, page titles and descriptions, grouped into sections. Free, instant, no signup.
robots.txt Tester
Test whether a URL is blocked for Googlebot, Bingbot, GPTBot or any crawler, see the deciding rule and line, and lint your robots.txt for common mistakes.