The Ultimate Robots.txt Guide

What is a Robots.txt File?

A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, use noindex directives or password-protect the page.

Robots.txt vs. XML Sitemaps

If you are confused about when to use robots.txt vs an XML sitemap, remember this simple rule:

• Robots.txt is for exclusion. It tells crawlers where not to go.
• XML Sitemaps are for inclusion. They tell crawlers where they should go.

Basic Syntax

The file consists of one or more rules. Each rule blocks or allows access for a given crawler to a specified file path in that website.

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /private/public-image.jpg

Sitemap: https://YOUR-DOMAIN.com/sitemap.xml

The User-agent Directive

This specifies which crawler the rule applies to. * means all crawlers. Specific crawlers include Googlebot, Bingbot, etc.

The Disallow Directive

This indicates the path that you do not want the user-agent to crawl. A trailing slash (/) means the directory and everything in it.

The Allow Directive

This is used to countermand a Disallow directive. For example, if you disallow a whole directory but want to allow one specific file inside it.

The Sitemap Directive

This is highly recommended. It tells all search engines where your XML sitemap is located. You can have multiple Sitemap directives in a single robots.txt file.

Common Mistakes & What NOT to Include

• Blocking JavaScript and CSS: Do not block the folders containing your site's JS and CSS assets. Google needs to render the page to understand it properly.
• Using robots.txt for security: Do not use robots.txt to hide sensitive information. The file is public, and anyone can read it to find your "secret" directories. Malicious actors often scan robots.txt files precisely to find these hidden paths.
• Conflicting Directives: If you Disallow a URL in robots.txt, but also include it in your XML sitemap, you send mixed signals to Google. Ensure your sitemap only contains Allowed URLs.
• Improper Wildcard Usage: Be extremely careful with wildcards (*). A poorly placed wildcard can accidentally block large sections of your site. Always test your rules thoroughly.

Best Practices for Formatting

To ensure your robots.txt file is processed correctly, follow these formatting guidelines:

• Capitalization Matters: The directives (e.g., User-agent:, Disallow:) are case-insensitive, but the URL paths are case-sensitive. /Admin/ is not the same as /admin/.
• One Directive per Line: Each rule must be on its own line. Do not combine multiple rules or paths on a single line.
• Use Comments Sparingly: You can use the # symbol to add comments. Comments are ignored by crawlers, but keeping the file clean and readable is best practice.

How to Test Your Robots.txt File

Before deploying changes to your production robots.txt file, it is highly recommended to test your rules. Even a small typo can lead to disastrous consequences for your SEO.

The most reliable way to test your robots.txt file is using the Robots.txt Tester tool within Google Search Console. This tool allows you to input specific URLs from your site and see exactly how Googlebot interprets your current robots.txt rules, highlighting any warnings or errors.