Skip to main content

SEO Tools · No account required

Robots.txt Tester & Checker

Use this free robots.txt tester and syntax checker to validate your file, test Googlebot crawl access, and find matching Allow or Disallow rules.

Instructions, examples and limitations ↓

What this tool does

This free robots.txt tester checks whether your pasted rules allow or block a URL for Googlebot or another crawler. The syntax validator flags malformed directives and unsupported extensions with line numbers. Review the winning Allow or Disallow rule, correct errors, and test again before publishing your file.

Enable JavaScript to use the local processing controls. Instructions and examples are available below.

Practical guide

Robots.txt Tester & Checker

By MultiZest · Updated

How to use this tool

  1. Open your site’s /robots.txt and paste its plain-text content, or load a saved robots.txt file.
  2. To test crawl access, enter the full page URL and a crawler product token such as Googlebot or Bingbot. Choose Validate syntax only to check the file without a URL.
  3. Select Check robots.txt. Review syntax diagnostics, the selected user-agent group and the winning rule with its line number. Fix errors and run the check again.
  4. After publishing changes, check the live /robots.txt response and use Google Search Console to inspect crawling and indexing separately.

Worked example

Disallow: /private/ blocks that directory, while Allow: /private/public-guide permits the longer matching path. At equal specificity, an Allow rule wins over Disallow.

How to test robots.txt with an Allow and Disallow example

Load the example in the tool, then test each URL below with Googlebot. These results describe the pasted file on example.com, rather than a live crawl.

User-agent: *
Disallow: /private/
Allow: /private/public-guide

Sitemap: https://example.com/sitemap.xml
URL to testExpected resultReason
https://example.com/private/notesBlockedDisallow: /private/ is the longest matching rule.
https://example.com/private/public-guideAllowedThe longer Allow rule takes precedence.
https://example.com/aboutAllowedNo Disallow rule matches this path.

What the robots.txt syntax validator checks

The checker reports missing colons, invalid user-agent tokens, rules outside a user-agent group, paths without a leading slash, and invalid sitemap URLs. It warns about whitespace or incomplete percent escapes in paths and directives that this tester does not use.

Empty Allow and Disallow values are valid and impose no restriction. Comments beginning with # are ignored. A clean syntax report applies to the supported directives; it does not verify your server response, every crawler extension, or whether your intended pages are accessible.

How Googlebot and wildcard rules are matched

Agent names are matched case-insensitively as exact product tokens; an exact group takes precedence over *. Multiple groups for that same token are combined. Paths remain case-sensitive, include the query string and support * wildcards and an ending $ anchor.

The longest matching literal path wins, with Allow preferred on a tie. Percent-encoded unreserved characters are normalized; reserved escapes remain escaped. Test Googlebot and other relevant crawler tokens separately because they may select different groups.

Common robots.txt errors and unexpected blocks

FindingWhat to check
Disallow: /This blocks every path for the selected group unless a more specific Allow rule matches.
Disallow: private/Add the leading slash: Disallow: /private/.
A rule before User-agentPlace the rule after the intended User-agent line. Ungrouped rules are ignored.
Noindex: /pageGoogle ignores this directive in robots.txt. Use a robots meta tag or X-Robots-Tag on a crawlable page.
Crawl-delayGoogle ignores this extension. Other crawler operators may support it.
Different capital letters in a path/Private/ and /private/ are different paths. Test the URL’s actual spelling.

Check the live file and Google Search Console separately

Publish the file at the root of the relevant origin, for example https://example.com/robots.txt. Subdomains and different protocols can have separate files. Confirm the server serves the intended plain text rather than an HTML error page, login page or outdated copy.

This independent tester works with the text you supply. Use Google Search Console’s robots.txt report for Google’s fetched file and URL Inspection for crawl and indexing diagnostics. This tool does not connect to your Search Console property or retrieve a live URL.

Crawl permission does not guarantee indexing

An allowed URL is not guaranteed to be indexed. A disallowed URL may still be known to a search engine from links. Robots.txt does not protect confidential pages and is not a substitute for authentication.

HTTP status, redirects, caching, robots meta tags and X-Robots-Tag headers are outside this local test. Paste up to 100,000 characters, load a plain-text file up to 100 KB, and use a test URL under 4,096 characters. Check the deployed file after every change.

Test a small set of URLs before publishing a rule change

A rule that solves one crawl problem can accidentally block something else. Before publishing, test one URL that should be blocked, one that should remain allowed and a public asset path relevant to rendering the page. Use the exact crawler token you are investigating rather than assuming every crawler selects the same group.

For example, a broad Disallow: /private/ can coexist with an Allow for a longer public path. Change the sample URL between the private and public pages and compare the winning rule. The purpose is to verify the intended exception, not merely to obtain one reassuring Allowed result.

Keep a record of the rule and the deployed response

Save the original robots.txt before editing and note which pages the change is intended to affect. After deployment, visit the origin’s actual /robots.txt and confirm that it serves the expected plain text. A local report can describe pasted rules correctly while the server still serves an older file.

Check Search Console’s fetched policy and URL Inspection when investigating Google’s behavior. Crawling, indexing and search appearance are different stages, and an unchanged ranking immediately after an edit does not prove the rule failed. Use the report’s line numbers to investigate the supplied file, then verify the separate live-state evidence.

Questions and troubleshooting

How do I check or test my robots.txt file?

Paste the content from your site’s /robots.txt, enter a full URL and select Googlebot or another crawler product token. Select Check robots.txt to see syntax diagnostics and the winning crawl rule. You can also validate syntax without a URL.

What is the difference between a robots.txt tester, checker and validator?

A tester evaluates crawl access for a URL and crawler. A syntax checker or validator identifies problems in the file. This tool combines both checks and reports line numbers; it does not verify the live server response.

Can I test Googlebot robots.txt rules here?

Yes. Enter Googlebot as the crawler product token. The tester selects matching Googlebot groups or falls back to User-agent: *. Googlebot ignores Crawl-delay and does not support Noindex in robots.txt.

Does an Allowed result guarantee Google indexing?

No. It only describes the supplied crawl rules. Indexing depends on additional technical and content factors. Use Search Console URL Inspection to investigate Google’s indexing state.

Does the tester retrieve my current robots.txt?

No. Paste its content manually or load a saved file. The test runs locally and does not confirm what your host currently serves.

Is robots.txt the same as robot.txt?

The standard filename is robots.txt, with an s. Publish it at the origin’s root, such as https://example.com/robots.txt; robot.txt is not the standard crawl-policy file.

Which URLs should I test after changing robots.txt?

Include paths that should be blocked, paths that should remain allowed, and relevant public assets. Test the crawler groups you actually use, then verify the deployed file separately.

Continue with a related tool

More detailed guides

Sources and further reading

Found an error or an unclear instruction? Report it. Read our editorial policy and privacy policy.