How to correctly format and structure a robots.txt file?

A robots.txt file should be formatted as a plain text file with one or more blocks of directives, each starting with a User-agent line specifying the targeted search engine bot, followed by one or more Disallow or Allow directives that control access to specific paths. Each directive must be on its own line, and the file can include comments starting with a # symbol. The file must be saved as robots.txt and placed in the root directory of the website.

Key formatting and structure points:

• Basic structure:

  User-agent: [bot-name or * for all bots]
  Disallow: [path to block]
  Allow: [path to allow]
  

• Multiple blocks: You can have multiple blocks for different user agents.

• Separate lines: Each directive (User-agent, Disallow, Allow) must be on a separate line to avoid parsing errors.

• Case sensitivity: Directive names (User-agent, Disallow, Allow) are case-insensitive, but the paths are case-sensitive.

• Comments: Lines starting with # are comments and ignored by bots.

• Wildcards and special characters:

  • * can be used as a wildcard in user-agent names and URL paths.
  • $ indicates the end of a URL pattern.

• Example:

  # Block all bots from /private/ directory
  User-agent: *
  Disallow: /private/
  
  # Allow Googlebot full access
  User-agent: Googlebot
  Disallow:
  
  # Block Bingbot from /no-bing/
  User-agent: bingbot
  Disallow: /no-bing/
  
  # Sitemap location (optional but recommended)
  Sitemap: https://example.com/sitemap.xml
  

• Sitemap directive: Including a Sitemap: line pointing to your sitemap helps search engines crawl your site more efficiently but is optional.

• Order of precedence: More specific rules override less specific ones. If rules are equally specific, the least restrictive rule applies.

This structure ensures search engines correctly interpret your crawling instructions and helps optimize crawl budget and SEO performance.