Robots.txt audit: the small file that can hide a whole client site.

A robots.txt audit takes ten minutes and still catches some of the most expensive problems we find on new client sites: a leftover Disallow: / from launch, blocked CSS, or a rule meant to remove pages from Google that does the opposite.

Key takeaways

  • A robots.txt audit checks that the file lets search engines crawl what matters and blocks only what should not be crawled.
  • Most damage comes from a few mistakes: Disallow: / left from launch, blocked CSS or JavaScript, wrong host, and 5xx errors.
  • Test with the Search Console robots.txt report and the URL Inspection tool, not by reading the file alone.
  • Disallow controls crawling, not indexing. To keep a page out of Google, use noindex and let Google crawl it.

A robots.txt audit is the first check in every technical review we run, because a single line can stop crawling across a whole site. Across 5+ years of technical SEO, including more than 600 US car dealership sites and 18 online stores with over 3 million products, the same handful of robots.txt mistakes keeps returning. This guide is the checklist our team uses when we deliver white label SEO for agencies. It sits inside the wider technical SEO audit checklist; here we go deep on one file.

What is a robots.txt audit?

A robots.txt audit is a review of the file that tells crawlers which URLs they may request. You confirm the file loads with a 200 status on every host, that its rules block only low-value URLs, that important pages and resources stay crawlable, and that nobody is using it to remove pages from Google, which it cannot do.

Google's introduction to robots.txt is clear on the purpose: the file manages crawler traffic and "is not a mechanism for keeping a web page out of Google". Everything in the audit follows from that sentence.

What are the most common robots.txt mistakes on client sites?

The usual suspects are a Disallow: / copied from staging, blocked CSS and JavaScript that pages need to render, rules on the wrong host or protocol, wildcard patterns that match more than intended, case mismatches in paths, and a robots.txt that returns server errors. Each one is quick to fix once someone actually looks at the file.

MistakeWhat happensFix
Disallow: / left after launchGoogle stops crawling the siteRemove it; protect staging with a password instead
Blocked CSS or JavaScriptGoogle cannot render the page properlyAllow resources that pages need to render
Rules on the wrong hostwww, a subdomain or http keeps old rulesCheck robots.txt on every host and protocol
Over-broad wildcardBlocks product or category URLs by accidentTest sample URLs against each pattern
Wrong case in paths/Shop/ rule does not match /shop/Match the exact case used in URLs
5xx on robots.txtGoogle pauses crawling, then relies on a cached copyFix the server response first

The technical rules are in Google's robots.txt specification. Paths are case-sensitive, rules apply only to the host, protocol and port where the file lives, and Google enforces a 500 KiB size limit and ignores anything beyond it. When rules conflict, Google applies the most specific rule and, in a tie, the least restrictive one. Errors matter too: a 4xx (other than 429) is treated as if no file existed, while a 5xx makes Google stop crawling for the first 12 hours and then use the last good version for up to 30 days. Google generally caches the file for up to 24 hours, so a fix is not instant.

The staging case deserves its own process. Launch-day handling of the development site lives in our staging site SEO guide; in the audit, you only confirm the production file is clean.

How do you test robots.txt with Search Console?

Open the robots.txt report in Search Console. It lists the robots.txt files Google found for the top hosts, the last check time, fetch status, size and any parsing issues. Then run key URLs through the URL Inspection tool to see whether each one is blocked. After a fix, request a recrawl of the file from the report.

Google's robots.txt report works only at domain level: a Domain property or a URL-prefix property without a path. In a Domain property it covers the top 20 hosts by crawl rate, and it keeps 30 days of version history, which is useful when a client says "nothing changed". Lines with problems are ignored and the rest of the file is used. Requesting a recrawl does not guarantee an immediate recrawl of the unblocked URLs. For single URLs, the URL Inspection tool shows whether crawling is allowed.

  • Fetch /robots.txt on every host (www, non-www, http, https, key subdomains) and note the status code.
  • Open the Search Console robots.txt report and read the fetch status and issues for each file.
  • Test 10 to 20 important URLs (home, top categories, top products or services) in URL Inspection.
  • Confirm CSS, JavaScript and image paths used by templates are not disallowed.
  • Check that the Sitemap: line points to the live, correct sitemap URL.
  • Compare against the previous version in the report history and the client's change log.

Disallow vs noindex: which one should you use?

Use disallow to stop crawling of URLs that waste crawl activity, such as internal search results or endless filter combinations. Use noindex to keep a page out of Google's results. Never combine them on the same URL: if robots.txt blocks the page, Google never sees the noindex rule, and the URL can still be indexed from links.

Google's noindex documentation says the page "must not be blocked by a robots.txt file" for noindex to work. For PDFs and other non-HTML files, send the rule as an X-Robots-Tag HTTP header. This is the mistake we see most in inherited audits: a client wants old pages gone, someone adds them to robots.txt, and months later they are still in the index as bare URLs. When the reason is duplication rather than removal, a canonical is often the better tool; see canonical tags for agencies.

Should robots.txt rules target specific crawlers?

Only when there is a clear business reason. Most client sites need one group for all user agents plus a sitemap line. Separate groups for named crawlers make sense when the client has decided to limit a specific bot, such as an AI crawler. Confirm each user agent token in the crawler's own documentation before adding it.

Google lists its crawlers and their user agent tokens in its overview of Google crawlers and fetchers. Remember that a crawler matches only the most specific group that names it, so a new group for one bot does not inherit the rules in the general group. Copy the rules you still want into it.

What should the robots.txt audit deliverable include?

Give the client a short section, not a lecture: the current file, each problem with the URLs it affects, the proposed new file, and a test plan. Add screenshots from the Search Console robots.txt report and URL Inspection. Most clients need a developer or platform setting to change the file, so state exactly who changes what.

Current vs proposed

Both files side by side, with each changed line explained in plain language.

Test results

Sample URLs with allowed or blocked status, before and after the change.

Follow-up

A recheck of the robots.txt report and indexing a week or two after release.

If the robots.txt problem is part of a bigger indexing drop, triage it with the page indexing report. Our white label technical SEO team delivers this audit under your brand, or you can book a 30-minute call.

Frequently asked questions

How often should we audit a client's robots.txt?

At onboarding, after every launch or platform change, and as a quick monthly check of the Search Console robots.txt report. Check that important pages and resources are not blocked and that staging rules were not copied to the live site.

Does robots.txt need a Sitemap line?

It is optional but useful. It helps crawlers find the sitemap even if nobody submitted it in Search Console. Add the full URL of the sitemap or sitemap index, and update it if the location changes. It is a simple way to make sure all search engines and crawlers can find the sitemap, including those not linked to Search Console.

Can a client edit robots.txt on Shopify or Wix?

Many hosted platforms allow edits through a template or setting. Check the platform's own help pages before promising a change. If editing is not possible, use page-level noindex tags or the platform's SEO settings instead. Test every change with Search Console's robots.txt report to make sure important pages stay crawlable.

Is a missing robots.txt a problem?

Not by itself. Google treats a 404 as if no file existed and crawls without restrictions. Add one if you need to block sections or point crawlers to the sitemap. Make sure the file never returns a server error, because Google may stop crawling the site until it can read it.

Do other search engines read the same file?

Yes. Bing and other major crawlers follow robots.txt, though support for some rules can differ. Test important rules in each search engine's own tools, and keep the file simple to avoid mistakes. Avoid relying on unsupported directives, such as noindex in robots.txt, which Google ignores.

Juan José Giraldo, senior white label SEO specialist

About the author

Juan José Giraldo is a senior SEO specialist with 5+ years of experience. He works on technical and local SEO for 600+ US car dealerships and supervises the SEO of 18 US e-commerce stores. He runs White Label SEO LATAM from Medellín, Colombia, and delivers white label SEO for agencies in the US, UK, Canada and Australia. Read more · LinkedIn.

Want these agency guides first? Choose White Label SEO LATAM as a preferred source on Google, and my articles are more likely to show up for you in Top Stories.

Follow on GoogleAdd as a preferred source

Sources

  1. Google Search Central, How Google interprets the robots.txt specification
  2. Google Search Central, Introduction to robots.txt
  3. Search Console Help, robots.txt report
  4. Google Search Central, Block Search indexing with noindex
  5. Search Console Help, URL Inspection tool
  6. Google Search Central, Overview of Google crawlers and fetchers

Ready to test a pilot client?

Book a 30-minute call. You get an honest read on fit, scope and price, with no obligation.