Skip to main content

SEO guide

How to check robots.txt without blocking important pages

By MultiZest · · 4 min read

A small robots.txt edit can have a broad effect. Disallowing a folder may solve one crawl problem while also hiding a page or resource you wanted a crawler to reach. The safest practical approach is to define the intended result, test several representative paths, and verify the file actually served after deployment.

The practical takeawayTest a blocked path, an allowed path and an important exception for each relevant crawler. A correct local report still needs a live-file check, and crawl permission is separate from indexing.

Write down what the rule is meant to change

Begin with one concrete intention: prevent a crawler from fetching a temporary export directory, for example, while keeping a public guide accessible. Save the original file and identify the exact URLs involved. A broad cleanup described only as “improve SEO” gives you no clear result to test.

Robots.txt is public crawl guidance rather than authentication. If the directory contains confidential files, protect them with an access-control mechanism. Listing a private path in a public policy does not make it secret or prevent a person from requesting it.

Check the selected crawler group first

User-agent: * supplies a wildcard group, but a crawler with its own applicable group may use that group instead. In this tester, exact crawler product-token groups take precedence, and repeated groups for the same token are combined. Enter Googlebot when that is the crawler you are investigating.

Do not assume that an Allowed result for one crawler applies to every other crawler. Test the tokens relevant to the site’s policy. A Googlebot-specific group and a wildcard group can legitimately produce different outcomes for the same URL, which is why the report names the selected group.

Use a blocked path and an explicit exception

With this illustrative file, /exports/customer-copy should be blocked and /exports/public-guide should be allowed by the longer matching rule. Test both full URLs and inspect the winning rule and line number. Then test /about, which has no matching Disallow rule and should remain allowed.

Paths are case-sensitive and can include query strings. Wildcards and an ending $ anchor change the scope of a match, so use examples that exercise them deliberately. A test that checks only the one blocked URL can miss an unintended public-page block.

User-agent: *
Disallow: /exports/
Allow: /exports/public-guide

Sitemap: https://example.com/sitemap.xml

Read diagnostics before trusting a green result

A missing colon or a path without its leading slash can cause a rule to be ignored. The remaining parsed rules may allow the URL even though the intended policy was to block it. Fix the reported syntax errors and run the same set of URLs again rather than accepting the crawl result in isolation.

Empty Allow and Disallow values are valid and impose no restriction. Unsupported directives need a different interpretation: Google ignores Crawl-delay and does not support Noindex in robots.txt. Use a crawlable page’s robots meta tag or X-Robots-Tag when the intention is to request exclusion from indexing.

Verify the deployed origin and response

Publish the policy at the relevant origin’s /robots.txt and visit that URL directly. Confirm that the response contains the intended plain text rather than a login page, HTML application shell or an old cached version. A subdomain or another protocol can have a separate policy, so inspect the origin used by the pages you tested.

Save the report alongside the intended file when a change affects important paths. The local tester analyzes the content supplied to it; it does not fetch the server response, follow page redirects or inspect HTTP headers. Deployment and caching need their own evidence.

Use Search Console for the next question

Use the robots.txt report to investigate the policy Google fetched and URL Inspection to investigate a page’s crawl and indexing state. An allowed page is not guaranteed to be indexed, while a disallowed URL may still be known from links. Those distinctions help prevent unnecessary edits when the real issue is elsewhere.

Review performance over comparable periods rather than judging a rule change from one quiet day. Keep the canonical page, visible content and internal links consistent while investigating. If a local decision and Google’s fetched state disagree, first establish which file and date each report describes.

  • Retain the original policy and a clear reason for each substantive change.
  • Test allowed pages and important exceptions as well as blocked pages.
  • Verify the live plain-text response after deployment.
  • Keep crawl rules, indexing directives and authentication as separate decisions.

Try the tools used in this guide

Sources and further reading

Continue reading