SEO guide
How to check robots.txt without blocking important pages
By MultiZest · · 4 min read
A small robots.txt edit can have a broad effect. Disallowing a folder may solve one crawl problem while also hiding a page or resource you wanted a crawler to reach. The safest practical approach is to define the intended result, test several representative paths, and verify the file actually served after deployment.
Write down what the rule is meant to change
Begin with one concrete intention: prevent a crawler from fetching a temporary export directory, for example, while keeping a public guide accessible. Save the original file and identify the exact URLs involved. A broad cleanup described only as “improve SEO” gives you no clear result to test.
Robots.txt is public crawl guidance rather than authentication. If the directory contains confidential files, protect them with an access-control mechanism. Listing a private path in a public policy does not make it secret or prevent a person from requesting it.
Check the selected crawler group first
User-agent: * supplies a wildcard group, but a crawler with its own applicable group may use that group instead. In this tester, exact crawler product-token groups take precedence, and repeated groups for the same token are combined. Enter Googlebot when that is the crawler you are investigating.
Do not assume that an Allowed result for one crawler applies to every other crawler. Test the tokens relevant to the site’s policy. A Googlebot-specific group and a wildcard group can legitimately produce different outcomes for the same URL, which is why the report names the selected group.
Use a blocked path and an explicit exception
With this illustrative file, /exports/customer-copy should be blocked and /exports/public-guide should be allowed by the longer matching rule. Test both full URLs and inspect the winning rule and line number. Then test /about, which has no matching Disallow rule and should remain allowed.
Paths are case-sensitive and can include query strings. Wildcards and an ending $ anchor change the scope of a match, so use examples that exercise them deliberately. A test that checks only the one blocked URL can miss an unintended public-page block.
User-agent: *
Disallow: /exports/
Allow: /exports/public-guide
Sitemap: https://example.com/sitemap.xmlRead diagnostics before trusting a green result
A missing colon or a path without its leading slash can cause a rule to be ignored. The remaining parsed rules may allow the URL even though the intended policy was to block it. Fix the reported syntax errors and run the same set of URLs again rather than accepting the crawl result in isolation.
Empty Allow and Disallow values are valid and impose no restriction. Unsupported directives need a different interpretation: Google ignores Crawl-delay and does not support Noindex in robots.txt. Use a crawlable page’s robots meta tag or X-Robots-Tag when the intention is to request exclusion from indexing.
Verify the deployed origin and response
Publish the policy at the relevant origin’s /robots.txt and visit that URL directly. Confirm that the response contains the intended plain text rather than a login page, HTML application shell or an old cached version. A subdomain or another protocol can have a separate policy, so inspect the origin used by the pages you tested.
Save the report alongside the intended file when a change affects important paths. The local tester analyzes the content supplied to it; it does not fetch the server response, follow page redirects or inspect HTTP headers. Deployment and caching need their own evidence.
Use Search Console for the next question
Use the robots.txt report to investigate the policy Google fetched and URL Inspection to investigate a page’s crawl and indexing state. An allowed page is not guaranteed to be indexed, while a disallowed URL may still be known from links. Those distinctions help prevent unnecessary edits when the real issue is elsewhere.
Review performance over comparable periods rather than judging a rule change from one quiet day. Keep the canonical page, visible content and internal links consistent while investigating. If a local decision and Google’s fetched state disagree, first establish which file and date each report describes.
- Retain the original policy and a clear reason for each substantive change.
- Test allowed pages and important exceptions as well as blocked pages.
- Verify the live plain-text response after deployment.
- Keep crawl rules, indexing directives and authentication as separate decisions.
Try the tools used in this guide
Use this free robots.txt tester and syntax checker to validate your file, test Googlebot crawl access, and find matching Allow or Disallow rules.
Robots & Sitemap Generator →Create robots.txt rules and an XML sitemap from your URLs. Configure crawler groups, download the files, and validate your rules before publishing.
Meta Tag Generator →Prepare page titles, descriptions, canonical URLs, and social-sharing tags with approximate previews.