Robots.txt Tester
Fetch a site's live robots.txt and test whether a specific URL is allowed or blocked for a given crawler, with the exact matching rule.
Test a URL
robots.txt
The site's live robots.txt will appear here.
The robots.txt tester fetches a site's live robots.txt and tells you whether a specific URL is allowed or blocked for a given crawler, and which rule decided it. Enter a URL and pick a crawler to confirm your important pages can be crawled.
How to use the Robots.txt Tester
- 1Enter the full URL you want to test.
- 2Choose the crawler, such as Googlebot or all crawlers.
- 3Click Test URL to fetch the live robots.txt and apply the rules.
- 4See whether the URL is allowed or blocked and the exact matching rule.
Why test robots.txt rules
A single wrong line in robots.txt can block an entire section of your site from search without any other warning. Because robots.txt uses pattern matching with wildcards and longest-match precedence, it is easy to block more than you intended. Testing a specific URL against the live file confirms exactly what crawlers can and cannot reach.
How allow and disallow rules are resolved
When multiple rules match a URL, the most specific rule wins, and an allow rule beats a disallow rule of the same length. This is why a broad Disallow can be safely overridden by a narrower Allow. The tester shows the matching rule so you understand precisely why a URL is allowed or blocked.
Per-crawler differences
Robots.txt can set different rules for different user-agents, so Googlebot, Bingbot and AI crawlers may each be treated differently. Testing per crawler lets you confirm that search engines can reach your content while any crawlers you intend to block are blocked, exactly as you planned.
Frequently asked questions
Is the robots.txt tester free?
Yes, it is free and needs no signup. Enter a URL and pick a crawler to test.
How does robots.txt decide allow versus block?
The most specific matching rule wins, and an allow beats a disallow of equal length. The tester shows the exact rule that applied to your URL.
Why is an important page being blocked?
Usually a broad Disallow rule matches it. The tester reveals the matching rule so you can add a narrower Allow or adjust the Disallow.
Can I test different crawlers?
Yes. You can test against Googlebot, Bingbot, common AI crawlers, or all crawlers, because robots.txt can set different rules per user-agent.
Does an allowed URL guarantee indexing?
No. Robots.txt controls crawling. A page also needs to be indexable, with no noindex tag, and worth indexing, to appear in search.
Want the full picture?
OneClickSEO tracks rankings, audits your site, monitors backlinks and your visibility across every AI search engine, all in one place.