One URL. Every rule that touches it.
The deciding line, the rules it beat, and a lint of the whole file.
Standards-based evaluation
0 ptsApplies RFC 9309 exactly as Google documents it: most specific user-agent group, longest matching path, Allow wins ties, wildcards and $ anchors.
- Disallow: / under User-agent: *
- No Sitemap: line
- Crawl-delay ignored by Googlebot
Deciding line
0 ptsShows the single rule that decides the verdict, its line number, and every other rule in the group that matched but lost on length.
- OAI-SearchBotOpenAIsearch
- Claude-UserAnthropicbrowse
- PerplexityBotPerplexitysearch
- Google-ExtendedGoogletrain
30+ crawlers at once
0 ptsAfter the crawler you picked, the same path is evaluated for Googlebot, Bingbot, every major AI crawler and the SEO tool bots.
- Meta tags
- Headings
- Images
- Schema
- Mobile
File lint
0 ptsFlags a wildcard Disallow: /, missing Sitemap lines, blocked CSS and JavaScript, duplicate groups, unsupported directives, BOMs, oversize files and paths that can never match.
Sitemap declarations
0 ptsValidates that every Sitemap: line is an absolute URL, since relative ones are silently ignored.
Annotated source
0 ptsThe whole file with line numbers, the deciding line highlighted, so you can fix it without guessing.
One character can hide a section.
From every search engine.
robots.txt is the one file where a single character removes a section of your site from every search engine. Most mistakes are invisible until traffic drops.
The longest rule wins, not the first
Crawlers do not read robots.txt top to bottom. The longest matching path decides, and Allow beats Disallow on ties. That is why an Allow rule added "below" a block often does nothing.
Groups shadow the wildcard
If a crawler has its own User-agent group, the wildcard group does not apply to it at all, even if the specific group is empty. Bots you named in 2023 may be ignoring rules you added in 2026.
Blocking assets breaks rendering
Google renders pages to rank them. A Disallow on your CSS or JavaScript folder makes the page look empty, which is worse than not blocking anything.
Fix what costs traffic first. Then tidy the rest.
- 1
Test the URLs that matter
Start with your homepage, a product or article page, and any path you deliberately blocked. Test each for Googlebot and for the AI crawlers you care about.
- 2
Read the deciding line, not the file
The result names the exact line. If it is the wrong one, either narrow the Disallow path or add a longer Allow path for the section you want open.
- 3
Fix errors before warnings
A wildcard Disallow: /, a relative Sitemap URL, or a blocked assets folder cost traffic today. Crawl-delay and unknown directives are just noise.
- 4
Keep one group per crawler
Duplicate User-agent groups are handled differently by different crawlers. Merge them so the behaviour is predictable.
- 5
Add the sitemap and re-test
Declare your XML sitemap with an absolute URL, then re-run the tester for the same URLs to confirm nothing regressed.
Stop finding crawl problems by hand
SiteLift audits robots.txt, sitemaps and indexability on a schedule, publishes AI-ready content, and tells you the moment a deploy blocks a crawler.
No credit card required
Frequently Asked Questions
Everything you need to know about the robots.txt tester.
More free tools
AI Bot Access Checker
See whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 14 other AI crawlers are allowed or blocked by your robots.txt, with the rule that decides it.
Sitemap Validator
Validate any XML sitemap: file structure, URL counts, lastmod freshness, live status sampling, duplicates, cross-host URLs and robots.txt declaration.
AI Readiness Checker
Check whether AI crawlers can read your site, whether you publish llms.txt, and how well your content is structured to be cited by AI assistants.